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Abstract 


This  exploratory  research^-aimed-to  assess  the 

.  . S'  V 

feasibility  of  creating  a  decision  support  system^ to  aid  “the?  & 
program  manager  in  determining,  the  evaluation  criteria  to  be 
used  in  a  Source  Selection.  The  topic  was  approached  in 
general  terms  with  a  review  of  the  DOD  and  USAF  regulations 
and  other  literature  pertaining  to  source  selection 
evaluation  criteria,  and  also  a  brief  review  of  decision 
making  processes  and  a^dei-lanm  support  ova. tern. framework.. 

A  dataset  of  81  previous  source  selection  plans  from 
Air  Force  Systems  Command,  Aeronautical  Systems  Div  iaioi^was 
examined  with  the  aid  of  a  commercially  available 
microcomputer~based  database  management  system  to  see 
whether  previous  experience  of  selection  of  evaluation 
criteria  could  be  incorporated  into  a  DSS.  A  heuristic 
methodilag^was  developed  which  can  provide  an  indication  to 
the  program  manager  of  the  criteria  areas  he  should  consider  for 
use  in  his  source  selection,  based  on  Wh^historical  data. 

It  involves  describing  the  system  under  procurement  in  terms 
of  specific  attributes  of  the  system,  and  using  the 
associations  observed  between  those  attributes  and  criteria 
within  the  historical  dataset  to  predict  the  likely  criteria 
for  the  new  system. 
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The  research  presented  only  the  general  basis  for  a 
DSS  and  further  research  is  required  to  establish  the 
validity  of  the  methodology  and  implement  a  DSS  in  any 
particular  operational  situation. 
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THE  FEASIBILITY  OF  A  DECISION  SUPPORT  SYSTEM 
FOR  THE  DETERMINATION  OF 
SOURCE  SELECTION  EVALUATION  CRITERIA 


I.  Research  Proposal 


Introduction 


Air  Force  Logistics  Doctrine,  AFM  2-18,  divides 
the  Logistics  System  into  the  four  interdependent  subsystems 
of  Requirements  Determination,  Acquisition,  Distribution,  and 
Maintenance.  The  acquisition  subsystem  is  often  described 
(2:9-2)  in  terms  of  an  acquisition  cycle  consisting  of  the 
folowing  phases: 

a.  need  identification,  review  and  validation, 

b.  conceptualization, 

c.  demonstration  and  validation, 

d.  engineering  development,  and 

e.  deployment  (which  leads  back  to  further 
need  identification). 

The  demonstration  and  validation  phase  - 

includes  extensive  studies  and  analysis  of 
alternatives  [of  proposed  systems]  and  may  include 
development  of  prototypes  or  other  hardware  with 
subsequent  test  and  evaluation  of  the  products. 
Competing  contractors  respond  to  the  [Air  Force's] 
request  for  proposal  (RFP),  detailing  their  approach, 
costs,  schedule,  management  plans,  additional  options 
and  various  other  information.  The  Air  Force,  through 
a  process  called  source  selection,  narrows  the 
competition  to  the  most  promising  options  and  signs 
contracts  for  design  development,  prototype 
development,  or  both  [2:183], 
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The  aims  of  the  source  selection  process  are 
detailed  in  Department  of  Defense  Directive  (DODD)  4105;62 
as  follows: 

The  prime  objectives  of  the  process  are  to;  (a) 
select  the  source  whose  proposal  has  the  highest  degree 
of  realism  and  credibility  and  whose  performance  is 
expected  to  best  meet  Government  objectives  at  an 
affordable  cost;  (b)  assure  impartial,  equitable,  and. 
comprehensive  evaluation  of  competitors*  pr-bpdsals-and* 
related  capabilities;  and  (c)  maximize  efficiency  and 
minimize  complexity  of  solicitatibn,  evaluation  and  the 
selection  decision  [40:2]. 

The  source  selection  process  is  managed  by  a 
Program  Manager  (PM)  under  the  direction  of  a  Source 
Selection  Authority.  The  Source  Selection  Authority  may  be 
as  high  as  the  Secretary  of  Defense  for  major  weapon  systems 
or  as  low  as  product  division  commanders.  The  selection  is 
carried  out  by  evaluating  contractors’  offers  against 
standards  called  Source  Selection  Criteria,  which  are 
detailed  in  a  pre-evaluation  prepared  Source  Selection  plan. 

The  source  selection  criteria,  and  the 
determination  of  those  criteria,  is  the  area  of  concern  for 
this  research  effort. 

The  Problem 

Much  criticism  is  levelled  at  the  source  selection 
procedures  because  of  the  time  involved  in  completing  the 
process.  Turner  (36:1)  quotes  Congressional  committee 
reports  which  call  for  the  "need  to  improve  practices  and 
reduce  delays  in  the  source  selection  process."  Among  the 
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reasons  for  concern  over  the  time  spent  in  the  source 
selection  process  is  the  high  cost  of  the  activity.  The 
variable  cost,  which  is  a  function  of  time,  not  only 
includes  the  element  of  the  salaries  of  highly  qualified 
personnel,  but  also  the  opportunity  cost  of  the  loss  of 
those  personnel  by  the  other  activities  from  which  they  have 
bee.  drawn,  the  opportunity  cost  of  delays  in  introduction 
of  the  weapon  system,  inflation  driven  price  escalations, 
and  so  on. 

Notwithstanding  the  objectives  of  the  source 
selection  process  discussed  previously,  and  mindful  also  of 
the  potentially  high  cost  of  an  inadequate  source  selection 
decision,  any  contraction  of  the  duration  of  the  process 
through  improved  efficiency  or  techniques  will  result  in 
substantial  cost  savings  and  is  a  topic  worthy  of  pursuit. 
The  problem  then,  is  the  high  time-dependant  cost  of  the 
source  selection  process  and  the  broad  aim  of  this  research 
effort  is  to  reduce  the  duration  of  the  source  selection 
process  consistent  with  maintaining  the  quality  of  the 
source  selection  decision. 

The  Research  Question 

The  standards  against  which  the  contractors’ 
proposals  are  evaluated  are  contained  within  the  Source 
Selection  Criteria.  The  validity  of  the  evaluation  is 
entirely  dependent  on  the  validity  of  the  criteria  since 
this  is  the  only  measure  of  the  worth  of  each  proposal. 


3 


Inappropriately  selected  criteria  renders  the  evaluation 
procedure  Invalid  before  it  even  starts,  and  an  invalid 
source  selection  is  not  only  very  costly  and  time-wasting, 

but  may  be  catastrophic  for  the  entire  project.  Thus  the 

* . . .  . ./ 

selection  of  the  most  suitable  criteria  for  a  given 
acquisition  is  a  vital  element  of  the  source  selection 

process. 

Selection  of  the  most  appropriate  criteria  first 
involves  an  indepth  study  by  the  program  manager  of  the 
weapon  system  characteristics,  then  an  identification  of  those 
areas  of  the  production  system  which  will  have  a  significant 
impact  on  the  success  or  failure  of  the  project  and  on  the 
operational  value  of  the  weapon  system  itself. 

A  complicating  factor  however,  is  that  the 

T 

personnel  employed  on  source  selections  often  have  little 
previous  source  selection  experience  and  may  be  approaching 
a  difficult  task  with  but  a  vague  sense  of  direction. 

Pingel  (29:12-13)  cites  such  situations  as  increasingly 
common  in  U.S.Navy  acquisition  activities,  and  Aeronautical 
Systems  Division  (ASD),  Air  Force  Systems  Command,  Wright- 

Patterson  AFB,  face  similar  problems  (22).  Even  with  the 

'< . •  .  .  .  y 

guidance  of  the  several  publications  described  in  Section  II 
and  the  assistance  of  those  personnel  with  experience  that 
are  accessable,  some  time  will  be  lost  by  the  inexperienced 
project  manager  while  he  "feels  his  way"  and  brings  his 
knowledge  up  to  a  level  which  enables  him  to  intelligently 
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attack  the  problem.  The  selection  of  the  evaluation 
criteria  is  a  particularly  difficult  job  for  the 
inexperienced  and  a  potential  source  of  future  problems  for 
the  source  selection  process  and  the  program. 

The  research  question  then  is:  Is  a  decision 
support  system,  designed  to  aid  the  project  manager  in  the 


determination  of  source  selection  evaluation  criteria, 
feasible? 


Background  ",:*t 

The  extent  of  criticism  of  the  source  selection 
process,  the  high  dollar  values  of  modern  weapon  systems, 
the  relative  infancy  of  the  source  selection  procedures,  and 
the  attraction  of  the  source  selection  activity  as  an  avenue 
of  litigation  by  unsuccessful  offerers  are  among  factors 
which  have  led  to  much  research  directed  toward  improving 
the  process. 

Turner  (36)  examined  the  techniques  and  methods 
used  by  each  of  the  three  U.S.  Armed  Services  to  ’’analyse 
their  salient  differences  [and]  to  identify  and  evaluate 
unique  or  innovative  source  selection  techniques  that  could 
be  useful  to  the  other  services."  Peters  (27)  evaluated  the 
source  selection  procedures  within  the  Air  Force  for  major 
modifications  with  a  view  to  identifying  and  correcting 
problems  and  "provide  an  alternative  to  Source  Selection 
Procedures . " 
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Closer  to  the  topic  of  this  research  effort, 

Helmer  and  Taylor,  in  their  Conceptual  Model  for  Evaluating 

Contractor  Management  During  Source  Selection  (21), 

developed  a  framework  for  evaluating  a  contractor's  proposal 

against  one  possible  area  of  selection  criteria.  Looking  at 

the  actual  decision  methodology,  Pingel  (29)  proposed  a 

"System  for  the  Use  of  Evaluation  Factors  in  the  Source 

Selection  of  Service  Contractors.”  Barclay  and  Nido  (4) 

addressed  the  decision  process  by 

identifying  a  model  of  the  source  selection  process,  as 
used  in  Aeronautical  Systems  Division,  Air  Force 
Systems  Command, ...[to]...  provide  a  basis  for  better 
understanding  the  quality  of  decision  information 
provided  by  the  process  and  form  a  framework  for 
improving  the  source  selection  process. 

In  their  report  (4; 11-16)  they  enumerated  on  other  works  in 

the  decision  area  by  Beard  (7),  Dawes  (14),  Dycus  (16),  Lee 

(24),  Milligan  (25),  Peterson  (28),  Simon  (35),  Waid 

(47)  and  Williams  (48). 

However,  a  search  of  the  literature  has  failed  to 
surface  any  research  specifically  concerned  with  the 
selection  of  the  most  appropriate  evaluation  criteria  for  a 
particular  procurement  activity  (apart  from  general  guidance 
given  in  Department  of  Defense  and  service  publications  as 
discussed  in  Section  II).  Likewise,  studies  ignore  how  to 
best  assist  the  project  officer  to  expediently  determine 
criteria  and  confidently  assess  their  quality  in  his 
particular  circumstances. 


The  Scope  of  the  Research 

This  research  effort  will  attempt,  to  provideone 
answer  to  the  question  of  how  the^pijo^ect,  officje^ ;«a^. be^-r 
assisted  in  the  selection  of  evaluation  criteria;  by  -* 
analysing  historical  data  from  completed  sour}ce  >selections 
with  the  view  to  ascertaining,  tjhe  feasibility  of  creating  , an 
information  system  which,  will;  serv'd, -as^a, .form .■i(of',j'cor|Ppr)ate,rt 
memory”  for  the  p.r0gram<  managerls  task v pf ._fcreating;  -thie, 
source  selection  evaluation  criteria.  The  work  will,  aim;  to 

determine  whether  a  store  of  historical  data  can  be  compiled 

,  ’  * 

and  processed  to  guide  the  inexperienced  project  officer  on 
the  track  towards  confident  selection  of  sound  evaluation 
factors,  tailored  to  the  peculiar  circumstances  of  his  : S,:,- 
project.  , ,  ;  ..  ‘ 

A  point  regarding  the  aim  of  this  research  should 
be  made  clear.  The  proposal  is  not.  to  develop,  a -system 
which  will  select  evaluation  criteria  or  in  any  way  relieve 
the  project  officer  of  the  responsibility  of  thoroughly 
investigating,  analysing  and  understanding  the 
characteristics  of  the  system  with  which  he  is  dealing  and 
then  determining  which  criteria  are  most  appropriate  to  that 
system.  Rather,  the  ultimate  goal  is  to  provide  a  tool  for 
the  project  officer  which  will  reduce  the  pain  of  the  first 
flounderings  in  the  search  for  the  evaluation  criteria  and 
result  in  a  more  timely  and  better  result.  An  operational 
database  or  software  package  will  not  be  produced  because, 


as  will  be  discussed  in  Section  Ill^ian  operational  package 
should  be  designed  from  the  base  tip  in  'conjunction  with  the 
several  office  automation  systems  that  are  currently  under 
development.  This  research  is  seen  more  as  an  exploratory^ 
feasibility  study  designed  to  establish  a  basis  for  the 
inclusion  of  source  selection  evaluation  criteria  in  project 
office  management  information  systems  to  ease  the  burden  of 
work  often  placed  on  the  inexperienced  project  officer" 
because  of  the  lack  of  good  corporate  memory  in  this  area. 

The  data  for  the  Study  will  be  taken  from  Source 
Selection  Plans  of  source  selections  undertaken  by  ASD. 

While  any  conclusions  drawn  from  examination  of  the  data 
will  thus  only  be  strictly  applicable  to  that  scenario,  the 
review  of  literature  concerning  the  source  selection  process 
given  in  Section  II  will  indicate  that  if  a  sufficiently 
general  approach  to  the  types  of  program  and  selection 
criteria  is  adopted,  there  is  sufficient  similarity  in 
procedures  Defense  -  wide  for  the  principles  to  be  valid  for 
source  selections  conducted  by  most  other  defense  agencies. 

Summary  and  Preview 

This  section  has  briefly  introduced  the  research 
setting,  the  nature  of  the  problem,  the  question  to  be 
answered  and  the  scope  of  the  research.  The  following 
sections  will  explain  in  detail  the  entire  research  effort. 
Section  II  will  present  a  review  of  the  literature 
pertaining  to  the  source  selection  process  with  emphasis  on 
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the  nature,  need  for,  and  importance  of  source  selection 
evaluation  criteria.  Section  III  will  treat  the  methodology 
used  throughout  the  research  by  initially  reviewing  decision 
support  systems  in  general,  the  decision  process  and  support 
of  that  process,  the  nature  of  the  decisions  for  source 
selection  criteria  determination  and  a  suggested  model  of 
those  processes.  The  same  section  will  then  describe  the 
data  used  for  the  research,  the  data  manipulation  tools  used 
and  the  development  of  a  heuristic  model  for  the  support  of 
criteria  determination  decisions.  Section  IV  will  present 
the  findings  derived  from  the  processing  and  examination  of 
the  data  to  determine  the  feasibility  of  using  the  heuristic 
model  as  the  core  of  a  decision  support  system.  Finally, 
Section  V  will  summarize  the  research  and  draw  conclusions, 
together  with  recommendations  for  further  research  in  the 
field. 


Q 
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'  v  II«  Literature  Review 

Introduction  " 

This  section  will  briefly  describe  the  Source 
Selection  Process,  concentrating  mainly  on  the  source 
selection  criteria.  The  official  Defense  publications  which 
relate  to  Source  Selection  will  be  explored  and  thpse' 
sections  which  give  direction  and  guidance  on  the  area  of 
source  selection  evaluation  criteria  will  be  highlighted. 

However,  the  review  of  this  area  bf  the  literature 
will  be  confined  to  Department  of  Defense  and  United  States 
Air  Force  publications.  The  justification  for  this  approach 
is  the  statement  contained  in  Department  of:  Defense 
Instruction  5000.2,  ftbloi  Systems  Acquisitions  C3S).  that 
"It  is  the  policy  of  the  Department  of  Defense  to  provide 
uniform  procedures  for  the  major  systems  acquisition 
process,"  coupled  with  the  conclusion  drawn  by  Turner 
(Xl:34)  that 

the  Air  Force  tends  to  try  to  cover  "all  bases"  with 
extreme  levels  of  detail  in  its  procedures  while  the 
Navy  gives  broad  general  guidance  in  most  areas  and 
leaves  the  buying  organization  to  come  up  with  their 
own  procedures.  The  Army  has  mixtures  of  both  with 
little  guidance  from  Headquarters,  DA,  but  greater 
detail  from  AMC. 

Similarly,  Babin  (3:6)  observed  that  "among  the 
DOD  components,  the  Department  of  the  Air  Force  appears  to 
have  the  greatest  amount  of  regulatory  material  specifically 
devoted  to  the  Source  Selection  Process." 


Thus,  a  coverage  of  the  information  presented  in 
that  subset  of  the  total  Defense  documentation  should  be 
sufficient  to  give  an  adequate  survey  of  this  area  of  the 
literature  and  yet  conform  to  the  time  constraints  of  this 
study.  For  the  reader  who  desires  information  on  source 
selection  related  publications  of  the  other  Services  Turner 
(36)  gives  a  good,  though  dated,  survey  and  Babin  (3)  adds 
a  brief  resume  of  civilian  publications. 

Regulatory  Publications 

The  prime  document  covering  U.S.  Government 
procurement  processes  is  Department  of  Defense  Directive 

5000.1,  Major  Systems  Acquisitions  (39)  and  the  second 
priority  document  is  Department  of  Defense  Instruction 

5000.2,  Major  Systems  Acquisitions  Procedures  (38)  each  of 
which  detail  mandatory  policy  concerning  the  acquisition 
process  for  procurements  designated  as  major  systems. 

The  subsystem  of  the  acquisition  process  which  is 
termed  Source  Selection  is  governed  by  Department  of  Defense 
Directive  4105.62,  Selection  of  Contractual  Sources  for 
Major  Defense  Systems  (40).  This  publication  gives  more 
detailed  policy  and  guidance  on  the  source  selection  process 
for  major  Defense  systems  than  does  DODD  5000.1  or  D0DI 

5000.2,  but  it  is  still  at  too  high  a  level  to  be  of 
significant  practical  assistance  to  the  program  manager. 
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The  major  USAF  document  is  Air  Force  Regulation  70- 
15  (AFR  70-15),  Source  Selection  Policy  and  Procedures 
(44),  which 

.  .  .  sets  policy,  assigns  authority  and 
responsibilities,  and  prescribes  implementing 
procedures  for  soliciting  and  evaluating  offerors' 
proposals.  It  al«o  provides  information  for  the 
selection  of  sources  for  development  and  production  of 
major  defense  systems,  subsystems*  and  components  as 
well  as  other  major  programs  or  projects  competitively 
procured  by  the  Department  of  the  Air  Force  [44:index 
page  1]. 

AFR  70-15,  together  with  all  other  relevant  defense 
regulations  (27:4),  implements  the  provisions  of  DODD 
4105.62.  Most  other  Air  Force  documents  either  amplify  and 
clarify  AFR  70-15  and  DODD  4105.62  or  delineate  policy  on 
situations  not  adequately  covered  by  those  regulations.  For 
example,  Headquarters  Air  Force  Systems  Command  (HQAFSC)  has 
produced  AFSC  supplement  1  to  AFR  70-15  (45),  and  AFSC 
Regulation  80-15  (46).  Further,  AFSC  Aeronautical  Systems 
Division  (ASD)  produced  ASD  Pamphlet  800-7  Source  Selection 
Guide  (43)  and  a  handbook  The  Source  Selection  Process 
(1). 

The  Source  Selection  Process 

AFR  70-15  formally  defines  the  source  selection 
process  as  "The  formalized  process  employed  in  competitive, 
negotiated  procurements  .  .  .  designed  to  insure  an 
impartial,  equitable,  and  economic,  evaluation,  and 
comparative  analysis  of  competing  offerors'  proposals  and 
their  capabilities  [44:4],"  and  places  dollar  limits  on 


*'•'  "**  "  j  j  *v  ■*  f’ii  ./  ^  j'  j,  **\  *5,  a  %  (  »  *>  j 

the  major  systems  procurements  with*  *Kich  it  'is  concerned, 
DODD'  4105.62  (38V^j  states  the  fliree  primary 
objectives  of  tlie  formal  source  seiectiW  ?p'rptess: 


(a)  select  the  source  whose  proposal  has  the  highest 
degree  of  .realism  and  credibility  and  whose 
performance  is  expected  to  best  meet  Government 
objectives  at  tan  if  fordable,  -cplst.;. 


(b),  assure  Imparttiaip  .$qui‘tab;Xe;».  'an.d.  .cpjnpreherisiv.e..  „ 
eypluation  of  dompetitprd*  proposals' and  relatei 
capabilities;.  ;and  > 


(c)  maximize  efficiency  and:  minimize  complexity  .of. 

solicitation,  evaluation  and  the  selection  decision. 

AFR  70-15  (44:3)  states  as  further  objectives  of 
the  source  selection  process  that: 

[The  process]  should  be  structured  to  properly 
balance  technical,  financial,  and  economic  or  business 
considerations  consistent  with  the  phase  of  the 
acquisition,  program  requirements*  and  business  and 
legal  constraints.  It  mutt  be  sufficiently  flexible  to 
accommodate  the  objectives  of  the  acquisition  and  a 
decision  must  be  compatible  with  program  requirements, 
risks,  and  conditions. 

Within  the  USAF  system  there  are  three  formal,  but 
distinctive,  methodological  approaches;  the  "Formal”  (three- 
step)  source  selection  organization;  a  "Streamlined”  (two- 
step)  organization;  and  the  "Four-Step"  organization.  The 
latter  was  instituted  into  the  Defense  Acquisition 
Regulations  with  the  issuance  of  the  September  1978  Defense 
Acquisition  Circular  (DAC-76-17)  (1:7-11). 

The  Formal  Source  Selection  Organization  as 


prescribed  by  AFR  70-15  is  a  three  tiered,  pyramidal 
structure  with,  at  the  apex,  the  Source  Selection  Authority 


(SSA),  an  "official  designated  to  direct  the  source 
selection  process,  approve  the  selection  plan,  select  the 
source(s)  and  announce  the  contract  award  [44:4]."  He  is 
served  by  a  Source  Selection  Advisory  Council  (SSAC)  which 
acts  as  his  staff  and  advises  him  on  the  findings  of  the 
Source  Selection  Evaluation  Board  (SSEB).  The  SSEB,  at  the 
base  of  the  pyramid,  is  a  group  of  functional  and  technical 
experts  who  "direct,  control  and  conduct  the  evaluation  of 
the  proposals  and  develop  summary  facts  and  findings 
[43:6]." 

The  organization  for  most  source  selections  follows 
the  same  basic  outline;  however,  there  is  sufficient 
flexibility  in  the  regulations  to  permit  adjustment  of  the . 
organization  to  suit  the  circumstances.  For  example,  where 
previous  experience  has  shown  that  the  SSAC  review  level  is 
not  necessary,  the  SSAC  and  SSEB  functions  can  be  combined 
into  a  single  Source  Selection  Evaluation  Committee  (SSEC) 
(the  Streamlined  Source  Selection  Organization)  (1:9; 

44:4;  43:7-9). 

Babin  (3:2,3)  concisely  describes  the  major  events 
in  the  source  selection  process  as: 

(a)  Identification  of  a  need  and  creation  of  a  plan  to 
satisfy  that  need. 

(b)  Communication  of  the  need  to  industry,  via 
solicitation.  This  is  usually  done  by  issuing  a 
Request  for  Quotations  (RFQ),  or  ...  a  Request  for 
Proposals  (RFP). 

(c)  Evaluation  of  proposals  and  the  establishment  of  a 
competitive  range.  Those  proposals  found  to  fall 
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outside  the  competitive  range  are  eliminated  from 
competition. 

(d)  The  conduct  of  written  and/or  oral  discussions  with 
those  offerors  within  the  competitive  range. 

(e)  The  evaluation  of  best  and  final  offers  following 
discussions. 

(f)  Selection  of  the  offer  most  advantageous  to  the 
Government. 

(g)  Award  of  the  contract. 

The  ’’Four-Step”  procedure,  which  is  applicable  to 

all  competitively  negotiated  research  and  development 

acquisitions,  differs  significantly  from  the  conventional 

method  described  previously  in  three  main  aspects: 

the  offeror's  technical  and  cost  proposals  are  not 
submitted  and  evaluated  simultaneously,  definitive 
contracts  are  not  negotiated  with  all  offerers  in  the 
competitive  range,  and  the  apparent  contractor  is 
selected  and  announced  prior  to  negotiation  of  a 
definitive  contract  [1:11]. 

In  particular,  the  offerors  do  not  learn  of  their  proposal’s 
deficiencies  until  during  Step  Four  when  they  are  disclosed 
and  resolved  as  part  of  the  negotiation  of  a  definitive 
contract. 

The  need  to  activate  the  source  selection  process  is 
triggered  by  submission  of  a  Justification  for  Major  System 
New  Start  in  response  to  the  submission  of  a  Required 
Operational  Capability  (ROC)  by  a  user  command.  Those 
activities  result  in  the  issuance  of  one  or  more  Program 
Management  Directives  (PMD)  (44:7).  The  PMD  guidance 
directs  the  Program  Office  in  its  preparation  of  the  Source 
Selection  Plan  (SSP),  the  key  planning  document  which 
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defines  the  organizational  approach  to  the  source  selection 
process  (43:3).  ^  : 

The  Source  Selection  Plan  (SSP)  gives  the  source 

*  ■  -  .  ^  ~  1 
,  ■  \  '  f 

selection  process  unity  and  purpose  of  direction.  It  lays 

I 

|  out  the  ground  rules  and,  in  particular,  defines  the 

|  1  •  - 
j  criteria  on  which  the  proposals  will  be  evaluated  and  the 

method  of  evaluation.  These  source  selection  evaluation 

criteria  are  the  heart  of  this  research. 

i 

Source  Selection  Evaluation  Criteria 

The  SSP  and  its  evaluation  criteria  will,  or  ideally 
should,  be  specifically  tailored  for  each  individual 
procurement  situation.  The  February  1984  issue  of  AFR  70-15 
(44:8)  explains  that: 

Award  will  be  based  on  an  integrated  assessment  of  each 
offerors’  ability  to  satisfy  the  requirements  of  the 
solicitation.  The  integrated  assessment  will  include 
evaluation  of  general  considerations  as  well  as  the 
results  of  the  evaluation  of  the  proposals  against 
specific  criteria. 

As  examples  of  the  general  considerations  it 

includes  past  performance,  proposed  contractual  terms  and 

conditions,  and  the  results  of  preaward  surveys.  It  further 

elaborates  on  evaluation  criteria  by  differentiating  between 

specific  and  assessment  criteria. 

The  assessment  criteria  relates  to  the  offeror's 
proposal  and  abilities.  They  typically  include  but  are 
not  limited  to  such  aspects  as  soundness  of  technical 
approach,  understanding  of  the  requirement,  compliance 
with  the  requirement,  past  performance  and  the  impact 
on  the  schedule. 
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Specific  criteria, 


on  the  other  hand,  relate  to  the 


program  characteristics..  Ty.pically  they  fall  into  five 


broad 


areas:(44:9)  >  W- 

a.  technical,  .  ; 

.  i  .'ll.:, 

b.  operational  utility 

c.  logistics;,  *  . 

d.  ;  management,,  ;  ;  .  , 

.  ^  {  V  .  ;  *  S 

e.  manufacturing,  >and-  - 


f.  test. 

The  specific  and  assessment  criteria  "provide  a 


matrix  that  identifies  and  interrelates  what.JLs  to  be 

evaluated  [44:8]."  The  assessment  criteria  :are  thus  a  means 

*  „  ' 

of  measurement  within  each  of  the  specific  criteria  areas, 
items,  and  factors.  An  example  of  the  prescribed  general 
format  for  the  matrix  of  the  evaluation  criteria  is  given  as 
Figure  1.  This  research  is  primarily  concerned  with  the 
specific  criteria  and  for  the  remainder  of  the  paper 
"criteria"  should  be  read  to  mean  "specific  criteria"  unless 
otherwise  qualified. 

The  specific  criteria  are  usually  broken  down  into 
"Areas,"  "Items,"  and  "Factors."  Areas  are  the  basic 
functional  disciplines  listed,  above  which  will  impact  on  the 
success  of  the  program  and  the  ultimately  procured  system. 
Items  are  a  more  specific  breakdown  of  the  areas  to  permit 
proper  analysis  and  evaluation.  Further  breakdown  reveals 
Factors,  or  even  Sub-factors  if  appropriate,  which  are  the 
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Figure  1.  Evaluation  Criteria  Matrix  Format  [44:19] 


criteria  or  standards  against  which  the  SSEB  makes  its 
evaluations.  An  example  of  the  breakdown  of  areas  into 
their  sub-parts  is  (taken  from  the  previous  issue  of  AFR  70- 
15  dated  16  April  1976): 

Area-Operatlons(identified  by  the  SSAC) 
Item-Maneuverability  (identified  by  the  SSAC) 
Factor-Turn  Radius(identified  by  the  SSEB) 
Factor-Excess  Power  (identified  by  the  SSEB) 
Item-Survivability 

Factor-Subsystem  redundancy 
Factor-IR  reduction 
Factor-Radar  cross  section 
Subfactor-Front  quarter 
Subfactor-Side  view 
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Proposals  are  evaluated  or  scored  by  a  variety  of 
methods,  and  considerable  latitude  is  given  as  to  which 
method  is  employed  in  any  given  instant.  The  two  extremes 
of  subjective  and  objective  evaluation  methods  are  narrative 
assessments  and  numerical  scoring  respectively.  A 
compromise  system  seeking  to  retain  some  of  the  advantages 
of  both  systems  is  a  color  code  assessment  system. 

Whichever  method  of  scoring  is  used  however,  the  proposals 
are  rated  by  comparison  against  standards,  not  by  comparison 
against  each  other(44:9;  1:60;  46:2-1). 

The  proposals  are  evaluated  against  the 
predetermined  standards  by  evaluators  on  the  SSEB  and  the 
results  are  submitted  as  a  SSEB  Evaluation  Report  to  the 
SSAC.  The  SSAC  analyses  the  SSEB  report,  considers  the 
relative  strengths,  weaknesses,  and  risks  and  presents  the 
final  SSAC  Analysis  Report  to  the  SSA.  The  SSA  is  the 
authority  which  makes  the  ultimate  source  selection  and 
announces  the  contract  award.  (44:4) 

Synopsis 

The  current  USAF  source  selection  processes  have 
been  examined,  but  to  fully  understand  them  requires  a 
background  history  to  gain  an  appreciation  of  how  and  why 
the  present  system  developed.  Turner  (36:5-20)  provides  a 
very  concise  but  comprehensive  description  of  Defense 
policy,  regulations  and  guidance  current  at  1975. 

Comparison  of  previous  regulations  since  1960  with  current 
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regulations  indicates  that  the  source  selection  process  is 
probably  far  from  maturity.  The  rate  of  change  of 
regulations  which  has  been  maintained  until  the  present  with 
no  apparent  levelling  out  suggests  that  there  may  still  be 
improvements  to  be  made  in  the  procurement  system. 

Further  evidence  of  the  continuing  evolution  of  the 
source  selection  process  was  furnished  by  an  initial  search 
through  decisions  and  unpublished  decisions  of  the 
Comptroller  General  of  the  U.S.,  from  1960  through  to  April 
1983,  by  FLITE  (Federal  Legal  Information  Through 
Electronics,  Denver,  Colorado),  which  revealed  seventy-one 
protests  directly  related  to  the  source  selection  process. 

A  majority  of  the  actions  occurred  in  recent  years  which  may 
suggest  the  source  selection  arena  is  becoming  more 
turbulent  rather  than  maturing  into  a  well  tried  and  stable 
system.  There  are  indications  that  the  problems  are  well 
recognized  (43:11),  but  the  field  is  ripe  for  further 
research  in  the  pursuit  of  a  better  process. 

Conclusion 

The  USAF  Source  Selection  Procedures  have  evolved 
over  many  years  and  with  the  experience  of  many  practical 
applications.  The  procedures  are  well  documented  in 
numerous  Department  of  Defense  and  Department  of  the  Air 
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Force  publications.  Additionally,  many  of  the  user 

organizations  have  added  their  own  manuals  to  the  wealth  of 

*  . / 

regulatory  and  guidance  materials. 

The  field  has  been  well  researched  and  will  most 
likely  continue  to  be  so  because  of  indications  that  the 
source  selection  procedures,  although  well  tried,  are  far 
from  perfect.  An  area  which  is  ripe  for  research,  and 
toward  which  this  thesis  effort  will  be  directed,  is  the 
determination  of  the  Source  Selection  Criteria. 


III.  Research  Methodology 


Introduction 

The  goal  of  the  research  methodology  is  to  devise 
a  way  to  examine  and  process  the  source  selection  evaluation 
criteria  actually  used  in  past  Source  Selections,  in  order 
to  provide  an  inexperienced  Program  Manager  with  an  aid  for 
determining  source  selection  evaluation  criteria. 

The  process  of  determining  the  evaluation  criteria 
for  the  selection  of  a  source  during  new  systems  acquisition 
is  a  complex  decision  process.  In  very  simple  terms,  the 
program  manager  must  research  the  system  undergoing 
acquisition,  determine  with  the  help  of  expert  advice  the 
key  variables  upon  which  the  selection  of  the  optimal  source 
depends,  and  then  operationally  define  and  weight  the 
significant  variables  which  become  the  evaluation  criteria. 

The  fact  that  this  is  a  complex  decision  process 

. ^ . .  . ,/ 

leads  one  to  consider  the  prospect  that  a  decision  support 
system  (DSS)  may  be  useful  to  the  program  manager.  A 
#  further  factor  common  in  source  selections  is  that  the 

responsible  program  manager  often  does  not  have  much,  if 
any,  experience  in  this  area  of  endeavor  prior  to  assignment 
to  a  program  office.  A  DSS  may  thus  be  useful  not  only  to 
support  the  decision,  but,  through  intelligent  and  practical 
design  it  may  be  developed  as  a  tool  to  lead  the 
inexperienced  program  manager  thr>ugh  the  investigative 
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procedures  necessary  to  arrive  at  a  position  from  which 
sound  decisions  are  possible.  This,  of  course,  assumes  that 
the  whole  process  can  be  modelled  in  a  structured  or  semi- 
structured  form. 

♦ 

Two  areas  of  required  initial  research  are  thus 
evident.  The  first  requirement  is  to  examine  the  theory  and 
science  of  decision  support  systems,  their  design, 
evaluation,  and  justification.  The  second  is  to  analyze  the 
source  selection  evaluation  criteria  determination  process 
and  model  it  in  a  form  supportive  of  the  design  and 
implementation  of  a  DSS.  The  success  of  the  venture  will  be 
dependent  on  the  satisfactory  modelling  of  the  system  and 
the  availability  of  DSS  technology  adequate  to  the  task  of 

. ;  dL.;.' 

constructing  a  useful  tool. 

Decision  Support  Systems  Overview  u-* c 

The  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  has  developed  a  Decision  Support 
Framework  for  Decision  Aid  Designers  (20).  Their  framework 
lists  the  steps  a  decision  aid  designer  should  ideally 
complete  in  the  full  development  and  implementation  of  the 
aid  or  aiding  system,  and  distinguishes  two  categories  of 
aids,  those  that  provide  information,  and  those  that  provide 
support  for  logically  and  rationally  evaluating  and 
integrating  information  in  making  a  decision.  The  first 
category  of  aids  is  termed  Information  Aids  and  the  second 
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category  Integration  Aids.  A  schematic  of  the  framework  is 
given  as  Figure  2. 

The  aim  of  a  decision  support  structure  such  as  just 
described  is  to  increase  the  range  of  a  decisionmaker's 
capabilities  to  make  a  rational  decision.  This  aim  implies 
the  acceptance  of  the  principles  that  decision  support  is 
used  when  human  judgment  is  a  critical  element  of  the 
decision  process*  and  decision  support  in  no  way  replaces 
the  decisionmaker  as  the  problem  solver.  Consequently  the 
aid  to  be  developed  here  will  be  a  tool  to  support  the  human 
judgment  and  decision-making  process. 

The  pivotal  point  of  the  framework  is  the  actual 
decision  to  be  eade.  An  analysis  of  tlf.at  decision  .provide, 
the  basis  for  definition  of  the  information  and  tools  needed 
to  make  the  decision.  These  determinations;  make  up  the 
Decision  Requirements.  The  quality  of  the  analysis  which 
results  in  the  formulation  of  the  decision  requirements  will 
determine  the  appropriatness,  quality  and  comprehensiveness 
of  the  final  decision  support  system  design. 

From  the  decision  requirements  will  emerge  the  basic 
form  of  the  system.  For  convenience  it  may  be  developed  in 
two  stages,  Information  Aids  and  Integration  Aids.  The 
Information  Aids  may  be  further  subdivided  into  two 
categories.  Data  Based  Aids  and  Calculation  Aids.  Data 
Based  Aids  make  available  to  the  decisionmaker  the  data  on 
which  the  decision  is  based;  automated  data  banks  providing 
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Framework  ,  (20:3), 


raw  or  summarized  data,  on  call,  in  accordance  with  either 
pre-determined  or  user~specified  criteria.  Calculation  Aids 
perform  mathematical  or  statistical"  computations  on  data 
drawn  from  the  data  base  or  other ,  sources,;  and  ;pr  esent  the 

'  ,  „  •  i  ^  '  ”  «.  i 

+  '  Tt  y  .  ^  £ 

results  to  the  decisionmaker.  Th<e  output  of.  both 
subdivisions  of  Informafioh  Aids,  regardlessi;-  o|  its  form,  is 
at  best  information  of  at  the  least  raw  idata. 

The  output  of  well  designed  Information  Aids  is 
often  appropriate  and  sufficient  for  the  decisionmaker  to 
arrive  at  a  satisfactory  decision.  But  just.  :as  often  the 
sheer  volume  of  information  made  possible  iby^  .and  available 

'  -  •  . i,  ; 

from,  the  aid  can  overwhelm  the  decisionmaker's  cognitive 
abilities  and  make  filtration  and  logical  evaluation  of.  the 
information  a  formidable  task. 

Integration  Aids  are  sets  .of  procedures  designed  to 
help  a  decisionmaker  logically  evaluate  and  Integrate  the 

‘  i  t 

information  provided  by  information  aids.  Their  composition 
and  organization  depends  on  the  characteristics  of  the 
decision  problem  which  hopefully  have  been  recognized  in  the 
Decision  Requirements  analysis.  They  may  serve  to  organize 
and  structure  information,  simplify  the  evaluation  and 
weighting  of  information,  help  overcome  judgmental  and 
cognitive  limitations  and  biases,  and  reduce  any  other 
psychological  difficulties  known  to  influence  the  class  of 
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decisions  with  which  they,  are  concerned*  ^Ultimately  the 
goal  of  every  Integration-  Aid*  is  to  help  the  decisionmaker 
arrive  at  a  logical,  rationaldecision.  " 

Having  observed -a,  decisionmaker  :arrive:  at  a  decision*! 
the  DSS  designer  should  evaluate  the  jsuppor,ti -given  to  the 
decisionmaker  at?  -least.- •jih/'teipgst  Ofat.hie't.jy.a'l-idity  and:  reliar 
bility  of.  the  ald,.*:its ^flexibility^;  and’  thedeg;ree  to  which 
it  led  to  an,  improved  :decisionv;  S,uch  ;evaluation  capacity 
should  also,  to  some  extent,  beibuilt  into  the:  support  system 
in  order  to  maintain  it  as  ah  aid  of  the  highest  calibre  and 
ensure  its  continued  validity,  :reliability.j,  and  flexibility* 

Decision  Concepts 

The  analysis  of  the  decision  process  and  the 

resultant  definition  of  the  Decision  Requirements  is  thus 
„  **  ‘  '  .  **•’'*/  *1- 

the  focus  which  gives  the  DSS  design  direction  and  is 

fundamental  to  the  success  of  the  support  system*  Keen  and 
Morton’s  Decision  Support  Systems;  An  Organizational 
Perspective  (23)  is  recommended  as  an  excellent  text  on 
which  to  base  such  an  analysis* 

Keen  and  Morton  identify  five  main  schools  of 
thought  on  conceptions  of  decisionmaking: 

a.  Rational  Man.  The  classical  conception  of 

decisionmaking  assumes  a  rational,  completely 
informed,  single  decisionmaker  using  a 
normative  decision  methodology  and  able  to 
evaluate  all  alternatives. 
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b,  Satisficing  Man.  H.A.  Simons*  (23:62)  , 

.  satisficing  decisionmaker  is;  »pr.acticaliy 

constrained  by  "bounded  rationality,”  and  so 
arrives  at  a-  "good;  enough"  solutionnby,  using 
heuristics.; 

c.  Organizational  Procedure.  Cyer.t  and  .March's 
(23:63)  organizational  process  view  .sees  the  - 

.  decision  as  a  function  of  the  Inter?* 
relationships:  among  components  of  an 
organization,  highlighting  a  simplified  and 
systematic  procedure  with  organizational 
structure,  mechanism*}  for  communication  and  co¬ 
ordination,  and  standard  operating  procedures. 
4*  Political  View.  The  political  view  represents 
the  decision  as  the  product  of  a  bargaining/- 
conflict  process  between  organizational 
subunits;  coalitions  of  individuals  with  vested 
interests,  where  power  and  influence  determine 
the  outcome  of  any  given  decision, 
e.  Individual  Differences.  The  individual 

decisionmaker  has  personality  and  style  which 
determines  his  approach  to  a  decision  problem, 
his  cognitive  ability,  and  the  degree  of 
subjectivity  of  his  personal  rationality. 
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These  five  conceptions  of  the  decision  process  range 
from  entirely  normative  to  entirely  descriptive,  but 
everyday  experience  of  decisionmaking  would  suggest  that 
most  decisions  in  reality  contain  elements,  of  more  than  one 
of  the  given  models.  Thus  the  analysis  of  the  decision 
situation  selected  for  support,  and  the  subsequent  model 
building  process,  should  adopt  an  eclectic  posture  and  avoid 
the  unwarranted  ommission  of  any  viewpoint  from  due 
consideration. 


Decision  Supportabilitv 

This  leads  one  to  consider  the  question  of  which 
categories  of  decisions  are  computer  supportable.  Simon 
introduced  two  general  categories  of  decisions  relevant  to 
this  problem,  programmed  and  nonprogrammed  decisions.  He 
explained  (34:5-6): 

Decisions  are  programmed  to  the  extent  that  they 
are  repetitive  and  routine  ,  to  the  extent  that  a 
definite  procedure  has  been  worked  out  for  handling 
them  so  that  they  don't  have  to  be  treated  de  novo  each 
time  they  occur.  Decisions  are  nonprogrammed  to  the 
extent  that  they  are  novel,  unstructured,  and 
consequential.  There  is  no  cut-end-dried  method  of 
handling  the  problem  because  it  hasn't  arisen  before, 
or  because  its  precise  nature  and  structure  are  elusive 
or  complex,  or  because  it  is  so  Important  that  it 
deserves  a  custom  tailored  treatment.  ...  By 
nonprogrammed  I  mean  a  response  where  the  system  has  no 
specific  procedure  to  deal  with  situations  like  the  one 
at  hand,  but  must  fall  back  on  whatever  general 
capacity  it  has  for  intelligent,  adaptive,  problem- 
oriented  action. 

Keen  and  Morton  substituted  the  names  structured  and 
unstructured  for  programmed  and  nonprogrammed  respectively. 
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and  introduced  a  third  category;  semistructured.  They 
considered  that  in  the  context  of  computerized  decision 
support,  the  structured  decision  would  be  well  Enough 
understood  to  be  given  in  its  entirety  to  clerks  and 
automated  through  the  computer,  thus  not  involving  a  manager 
at  all.  An  unstructured  decision,  on  the  other  hand,  is  not 
capab?  e  of.  being  programmed  or  appears1  to  be  so  because  it 
has  not  yet  been  examined  in  sufficient  depth.  In  either 
case  the  decision  is  made  by  the  manager  and  decision 
support  is  not  yet  possible.  The  category  of  semistructured 
decisions  covers  that  very  broad  area  between  the  two 
extremities  of  structured  and  unstructured  decisions,  where 
a  DSS  can  be  most  effective.  It  is  here  that  neither 
managerial  judgment  nor  a  computational  system  alone  is 
sufficient  to  provide  a  solution  because  the  first  cannot 
cope,  perhaps,  with  the  size  or  complexity  of  the  problem, 
and  the  other  cannot  provide  a  necessary  subjective 
analysis. 

A  final  word  is  appropriate  on  the  design  model  and 
the  system  process.  Decision  support  models  applicable  to 
the  semistructured  decision  category  must  differ 
substantially  from  the  optimization  algorithms  used  in  the 
structured  area.  The  best  models  will  probably  be  simple, 
small  and  informal,  eliciting  better  answers  than  currently 
achievable  rather  than  dangerously  non-sub jective  optimum 
solutions,  and  which  represent  a  manager's  concept  of  the 


key  interactions  of  environmental  variables.  Along  the  same 
lines,  the  users  should  ideally  be  involved  in  the 
development  of  a  DSS  in  order  to  create  a  tool  which  will 
most  useful  in  practice,  and  will  grow  according  to  a  user 
driven  evolutionary  process  throughout  its  useful  life. 

This  is  the  concept  of  bottom-up  development. 

Tackling  the  determination  of  the  Decision 
Requirements,  one  must  make  an  intuitive  assessment  of  the 
style  of  DSS  which  will  be  most  suited  to  the  decision 
situation  and  proceed  with  the  decision  analysis  on  that 
basis.  Again,  user  input  to  DSS  pre-design  at  this  point  is 
highly  desirable.  The  question  of  the  DSS  style  relates  to 
the  desirability  of  designing  a  DSS  based  on  a  normative 
model,  which  achieves  the  objectives  of  the  decision  process 
but  may  be  radically  different  from  the  present  decision 
process,  or  whether  a  model  based  on  descriptive  analysis  of 
the  present  process  is  preferable.  The  potential  payoff  of 
the  former  style  of  DSS,  if  such  a  model  is  possible,  may  be 
huge,  but  design,  development,  and  implementation  is  likely 
to  be  long,  expensive,  and  risky.  The  latter  type  of  DSS, 
which  is  very  little  distant  from  the  descriptive  system, 
represents  a  reinforcement  of  current  practices  with 
probably  a  relatively  low  payoff.  However,  implementation 
should  be  easy  and,  if  well  designed,  should  encourage  the 
evolutionary  growth  previously  discussed.  The  potential 
payoff  in  the  long  term  may  thus  be  high  and  the  likelihood 
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of  success  is  very  much  greater.  The  decision  support 
design  objective  selected  for  this  project  will  therefore  be 
to  create  a  tool  to  enhance  performance  largely  within 
existing  procedures,  and  the  decision  model  will  be 
descriptive  of  the  current  processes. 

The  Evaluation  Criteria  Determination  Process 

The  procedure  followed  by  a  program  manager  to 
determine  the  source  selection  evaluation  criteria  will  vary 
with  individual  preferences  from  manager  to  manager  and  also 
from  project  to  project  as  experience  with  source  selections 
for  similar  systems  varies.  The  approach  adopted  here  to 
cater  for  these  individualities  will  be  to  build  a 
descriptive  model,  based  on  discussions  with  practising 
professionals  in  the  field,  of  a  typical  but  contrived 
procedure  which  is  considered  representative  of  actual 
practice.  The  limitations  of  this  approach  are  recognized, 
but  in  the  interests  of  convenience  and  hence  practicality, 
it  will  serve  to  establish  the  validity  of  the  concept  of 
the  DSS,  which  is  the  prime  objective  of  this  effort.  A 
practical  DSS  destined  for  a  specific  installation  must,  of 
course,  be  designed  with  the  peculiar  requirements  of  that 
particular  installation  always  in  view.  A  diagrammatic 
overview  of  the  contrived  procedure  is  given  in  Figure  3, 
and  the  first  level  of  analytical  detail  in  Figure  4. 

The  Program  Manager's  search  for  the  evaluation 
criteria  to  be  used  in  a  given  source  selection  is 
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EVALUATION  CRITERIA  DETERMINATION 


undoubtedly  a  decision  process.  At  the  highest  level  of 
abstraction  the  entire  process  is  directed  by  a  question: 
’’What  are  the  appropriate  source  selection  evaluation 
criteria  for  this  acquisition?” 

The  first  and  most  basic  requirement  for  the  PM  is 
to  research,  examine,  and  analyse  his  system.  This  is 
fundamental  to  the  PM’s  raison  d'etre  and  to  the  success  of 
the  source  selection.  In  the  context  of  the  procurement  of 
a  sophisticated  and  complex  major  system,  the  PM  would  not 
of  course  be  expected  to  be  fully  intimate  with  every  detail 
of  the  system.  In  these  cases  (the  majority  of  cases)  one 
should  read  PM  to  mean  "a  team  of  experts  drawn  from  a 
balanced  selection  of  appropriate  fields.”  Similarly 
fundamental,  the  PM  will  need  to  be  fully  conversant  and 
comfortable  with  the  objectives  of  the  project  and  the 
environmental  inputs  which  may  impact  on  the  choice  and 
performance  of  a  contractor. 

The  environment  may  be  so  complex  and  dynamic, 
particularly  in  the  case  of  a  high  dollar  value  project  with 
strong  political  overtones,  that  the  PM  may  need  a  formal 
environmental  scanner,  or  in  other  cases,  perhaps  the 
experience,  intuition,  and  judgment  of  the  PM  may  be  a 
sufficient  environmental  corapensator/reactor. 

The  PM  must  then  select,  define,  and  weight  the 
criteria.  The  recognition  of  the  influences  and  relative 
importance  of  the  inputs  discussed  to  this  point  provide  a 
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basis  for  differentiation  among  criteria*  but  the  immediate 
problem  now  is  where  to  find  ah  initial  criteria  set  to 
catalyse  the  process.  A  brief  look  at  some  categories  of 
evaluation  criteria  may  be  helpful  at  this  stage. 

The  evaluation  criteria  typically  used  in  source 
selection  may  be  categorized  into  three  hierarchical  levels: 
a.  Compulsory  Criteria.  Certain  evaluation 

criteria  are  prescribed  by  regulation  and  must 
be  included  in  either  all  source  selections  or 
specifically  designated  source  selections.  For 

y 

the  selection  of  these  criteria*  the  decision 
is  out  of  the  hands  of  the  Program  Manager  (PM) 
cr  the  Source  Selection  Authority  (SSA). 
b  Recommended  Criteria.  Certain  other  evaluation 

criteria  are  strongly  recommended  by 
Regulations  or  official  handbooks.  Here  there 
is  some  measure  of  discretion  available  to  the 
PM  and  the  SSA. 

c  System  Related  Criteria.  The  remaining 

criteria  are  those  which  are  selected  solely  on 
the  basis  of  their  relevance  and  importance  to 
the  unique  system  being  procured. 

The  criteria  may  then  be  further  broken  down  into 
Areas*  Items  and  Factors  in  the  manner  described  in  Section 
II  and  AFR  70-15  (44:9,19). 
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While  the  Compulsory  and  Recommended  Criteria  as 
described  above  should  present  no- insurmountable'  problems  . 
for  the  PM  in  their,  inclusion  in  his  set  of  criteria,  their 
origin  and  authority  can  often  be  obsfcure  and  they  mayi:  v 
escape  the  attention  of  an  inexperienced  EM.,  Further,,  they: 
may  still  need  to  be  divided'  into.  Items  .or  Factors,  and  . 
require  some  analysis  for  their  precise  definition  in  a 
given  procurement  situation.  ,■ 

System  Related -Criteria  can  giva  the  PM  considerable 
trouble,  first  in  identification  as  significant  factors-, in 
the  contracting  process  or  system,  lifercycle*  and  secondly 
in  the  determination  of  their  validity  and  relative 
importance.  That  assertion  is  widely  supported.  Carnes?  j" 
(8:20),  for  example*,  concluded  during  a  Source  Selection 
case  study  that  "the  most  difficult  task  in  preparing  the 
Source  Selection  Plan  .  .  .  was  the  development  of  the 
source  selection  criteria.” 

There  appears  to  be  two  ways  to  attack  the  problem. 
One  is  to  try  to  arrive  at  the  criteria  by  intuition, 
extensive  research,  survey  of  expert  opinion,  trial  and 
error,  and  so  on,  but  this  method  usually  takes  considerable 
time,  is  risky,  and  is  inefficient  unless  the  PM  has  a  good 
deal  of  experience  in  similar  projects  or  acquisition  from 
which  to  draw.  The  alternative,  and  also  the  most  commonly 
used  approach,  is  for  the  PM  to  search  back  through  the 
files  of  previous  Source  Selections  until  he  finds  a  project 
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with  similar  characteristics  to  those  of  his  own  project  and 
then  uses  the  evaluation  criteria-  recorded'  ^therein  as  his 
starting  point.  Unless  abused  by  an  overly  ready  acceptance 
of  the  previous  criteria  as- they  stand  and  without  due 
regard,  to  the  peculiar  requirements  of  the  current 
situation,  then  this  procedure.  Is  relatively  fast  and 
useful.  ■' 

Having  determined  a  starting  set  of  criteria,  they 
can  then  be  compared  to  the  various  requirements' of*  the 
system,  the  objectives  of  the  project  and  the  environmental 
inputs.  The  criteria  can  be  modified  and  honed  by  an 
iterative  process  until  a  satisfactory  set  is  arrived  at. 

They  must  then  be  operationally  defined:  since  they  provide  ; 
the  standards  against  which  competing  proposals  will  " 

ultimately  be  measured.  Further-,  since  not  all  criteria 
will  be  of  equal  importance  they  must  be  ranked  and  weighted. 
This  is  typically  done  by  expert  opinion,  regulation,  and 
the  use  of  a  suitable  technique  such  as  value  analysis, 
value  building,  or  multi-attribute  utility  techniques. 

The  criteria  must  then  be  validated.  Validation  may 
be  simply  some  sort  of  review  and  approval  process  as  would 
be  provided  by  a  Source  Selection  Authority,  or  it  may  also 
include  a  more  rigorous  examination  using  sensitivity 
analysis  or  simulation.  Finally  the  evaluation  criteria  is 
written  into  the  Source  Splection  Plan  and  the  decision 
process  is  complete. 
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The  Decision  Model 


The  specific  portion  of  the  model  thus  far  described 
which  is  attractive  for .the  incorporation  of  decision 
support  because  of  its  Targe  number  of  inputs  and 
constraints  is  the  section  in  which  the  evaluation  criteria 
are  selected,  defined  and  weighted.  This  area  will  be 
examined  in  more  detail. 

If  the  assumption  is  maderthat  the  PM  will  review 
the  historical  records  in  search  of  the  latest  similar 
source  selection  as  a  starting  point  in  the  determination  of 
criteria,  then  Figures  5  and  6  show  a  typical  procedure. 

The  figures  contains  only  the  search  of  historical 
records  and  the  decision  process  for  use  or  rejection  of  the 
historical  data.  If  modular,  bottom-up  evolutionary 
development  is  accepted  as  a  valid  practical  approach  to  JDSS 
construction  then  this  small  section  of  the  'model  appears  to 
be  a  good  place  to  start. 

The  PM  searches  backward  through  the  files  of 
previous  source  selection  activities  until  he  finds  a  system 
which  is  comparable  with  his  own  project.  He  then  extracts 
the  evaluation  criteria  that  were  used  in  the  past  source 
selection  and  divides  them  into  the  subsets  of  Compulsory, 
Recommended,  and  System  Specific  Criteria  as  described 
previously.  The  Compulsory  criteria  may  be  recorded 
immediately  for  use  in  his  own  source  selection. 


Figure  5,  Criteria  Determination  Procedural  Flow 
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The  other  two  subsets  must  be  examined  in  the  light 
of  the  following  considerations: 

a.  Are  the  criteria  appropriate  for  this  system? 

b*  Do  the  criteria  conform  to  the  system 
objectives? 

c.  Do  the  criteria  conflict  with  environmental 
constraints  which  are  present  at  this  time  or 
forecast  for  the  relevant  future? 

If  a  criterion  does  not  pass  all  these  tests  then  it 
may  be  considered  for  its  potential  for  acceptance  if  it  is 
modified  to  counter  its  deficiencies.  It  will  either 
undergo  modification  and  retest  or  be  discarded.  Successful 
criteria  will  then  be  recorded  as  the  compulsory  criteria 
were. 

The  testing  process  for  the  non-compulsory  subsets 
is  divided  into  two  separate  channels  because  the  slightly 
different  nature  and  emphasis  of  the  two  categories  would 
normally  require  different  comparative  processes  and 
environmental  impact  considerations.  A  more  efficient 
decision  process  will  often  be  achieved  by  considering  them 
as  (subtly)  different  processes. 

A  starting  list  of  appropriate  criteria  now  exists 
to  which  should  be  added  further  criteria  which  are  either 
compulsory  or  recommended  criteria  and  which  the  PM  observes 
as  having  been  omitted  from  the  historical  data  or  system 
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specific  criteria  which  the  PM  develops  from  scratch.  This 
is  a  reversion  to  the  first  method  described  earlier  and 
could  take  considerable  additional  time.  The  final  list  of 
criteria  thus  obtained  would  then  be  subject  to  further 
scrutiny,  sensitivity  analysis,  etc  to  ensure  that  only 
important,  relevant  and  non  trivial  factors  were  included  in 
the  final  set. 

The  Support  Structure 

The  fact  that  there  have  been  a  multitude  of 
programs  and  therefore  Source  Selections  in  the  past, 
suggests  that  the  store  of  historical  data  available  is  very 
large.  This  would  be  true  except  for  two  constraints. 
Firstly,  because  of  the  sensitive  nature  of  source  selection 
records  and  the  requirement  to  safeguard  that  information, 
there  will  only  be  a  limited  quantity  retained  in  the  source 
selection  offices  and  thus  readily  accessible  by  the  PM. 
Secondly,  the  constraints  of  time  imposed  on  the  PM  means 
that  he  would  normally  not  be  motivated  to  search  further 
than  the  first,  or  maybe  second,  file  containing  suitable 
data. 

On  the  other  hand,  if  the  total  experience  obtained 
on  all  previous  source  selections  could  be  aggregated  and 
data  that  could  serve  to  assist  the  PM  on  his  particular 
project  be  presented  to  him  in  a  useful  form,  then  the 
beneficial  input  from  all  comparable  past  source  selections 
could  be  focussed  into  the  current  project.  Here  then  is 


the  first  part  of  a  DSS  module;  the  data  base,  an 
Information  Aid. 

However,  to  rest  with  just  the  data  base  and  a 
retrieval  structure  would  be  premature.  With  the  outputs 
gained  from  the  earlier  stages  of  the  evaluation  criteria 
determination  process  and  the  output  of  the  information  aid, 
the  incorporation  of  a  good  Integration  Aid  T^ould  produce  a 
very  effective  decision  support  module. 

The  benefits  would  be  the  assimilation  of  far 
greater  volume  of  historical  data,  the  ability  to 
incorporate  a  greater  number  of  environmental  inputs, 
probable  reduction  of  human  biases  during  criteria  testing, 
shorter  search,  processing  and  selection  time  and  less 
likelihood  of  not  considering  or  overlooking  an  important 
criteria.  Indeed,  the  effect  of  the  vastly  increased  base 
of  historical  data  may  reduce  the  need  to  search  extensively 
for  further  criteria  to  virtually  zero  so  long  as  the  new 
system  is  not  so  unique  as  to  be  revolutionary. 

The  Database 

The  database  to  be  used  for  the  research  is 
drawn  from  Source  Selection  Plans  for  previous  source 
selections  dating  from  1971  and  held  on  file  at  Aeronautical 
Systems  Division  (ASD),  Air  Force  Systems  Command,  Wright- 
Patterson  AF6.  The  database  thus  obtained  consists  of  81 
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sets  of  evaluation  criteria  from  acquisitions  covering  a 
broad  range  of  systems  for  which  ASD  was  the  program  agency. 

Limitations  of  the  data  sample  arise  primarily 
from  convenience,  expense,  and  data  sensitivity. 
Conveniently,  the  data  was  drawn  only  from  the  files  of  ASD 
even  though  there  are  numerous  other  agencies  which  conduct 
source  selections.  Such  a  restricted  division  of  the  total 
population  means  that  the  types  of  weapons  systems  included 
in  the  sample  will  not  be  representative  of  all  systems 
procured  by  DOD  or  even  the  USAF.  However,  the  range  of 
systems  included  in  the  data  set  will  be  adequate  to 
establish  the  feasibility  of  the  methodology  and  thus  will 
satisfy  the  objectives  of  the  research  as  detailed  in 
Section  I.  Consequently,  the  additional  time  and  expense 
involved  in  acquiring  a  broader  database  was  not  considered 
to  be  warranted. 

The  nature  of  the  Source  Selection  Process  makes 
any  information  drawn  from  the  files  of  previous  proceedings 
sensitive  to  some  degree.  Thus  the  information  used  for  the 
database  was  confined  to  that  information  which  was  made 
public  by  virtue  of  its  use  in  Requests  for  Proposals  (RFPs) 
during  the  acquisition  process.  Extension  of  the  research 
to  include,  for  example,  weightings  applied  to  evaluation 
criteria  would  have  required  the  research  to  be  classified 
and  access  to  the  data  may  have  presented  problems.  Those 
studies  may  be  undertaken  profitably  by  researchers 
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operating  in  a  rriore  appropriately  controlled  environment* 

Database  Management  System 

Manipulation  of  the  data  set  required  the  selection 
of  a  database  management  system  capable  of  recording, 
sorting,  searching,  and  reporting  data  in  an  appropriate  and 
convenient  mann<er.  Several  options  were  available. 

Broadly  speaking  the  alternatives  were  to  create  a 
new  database  management  system  from  scratch  using  a  high 
level  programming  language  such  as  Fortran,  Pascall,>  Basic, 
or  Simscript»  or  to  use  a  proprietary  database  management 
system.  Since  the  objective  of  this  effort  is  to  address 
the  concept  and  feasibility  of  a  decision  support  framework, 
the  latter/option  of  a  commercially  available  software 
package  was  selected  for  its  convenience  and  simplicity. 

For  follow-on  work  in  more  specific  design  situations  the 
selection  would  demand  a  reassessment  for  its  continued 
suitability.  The  number  and  variety  of  available  database 
management  systems  is  large  and  growing  rapidly.  They  range 
from  large  and  versatile  packages  requiring  mainframe 
computer  facilities  to  extremely  simple  electronic  filing 
systems  designed  for  use  on  the  smallest  personal 
microcomputers.  In  order  to  make  a  reasonable  choice  from 
among  the  myriad  of  alternatives  the  author  developed  the 
following  selection  criteria 

a.  Adequacy  of  database  capacity. 

b.  Ease  of  use. 
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c.  Time  to  learn  (since  the  author  did  hot  have 
operator,  familiarity  with  any  suitable 
package). 

d.  Data  manipulation  capability. 

e.  Report  presentation  capability. 

f.  Availability. 

g.  Convenience. 

The  system  which  best  satisfied  the  aggregate  of 
these  criteria  and  which  was  therefore  selected  as  the 
management  tool  was  an  assembly-language  relational  database 
management  system,  produced  by  Ashton-Tate,  Culver  City,  CA, 
called  dBASE  II. 

The  principal  advantages  of  dBASE  II  were  its 
availability  (it  is  a  very  popular  and  commonly  used 
software  package  for  microcomputers),  convenience  (it  could 
be  run  on  the  author's  personal  Apple  11+  microcomputer), 
ease  of  learning,  and  its  ability  to  output  reports  in 
suitable  format  for  use  with  Wordstar,  the  word  processor 
being  used  to  produce  this  thesis.  Furthermore,  dBASE  II  is 
already  in  use  in  the  contracting  and  acquisition  arena 
(6;  17),  in  program  offices  and  in  some  source  selection 
facilities  (5).  Thus  there  appeared  to  be  some  benefit  in 
remaining  with  a  cheap,  commonly  used,  and  familiar  system 
which  may  encourage  users  to  attempt  the  development  of 
decision  support  aids  even  in  times  of  stringent  budgetry 
constraint.  But  the  selection  criteria  used  are  appropriate 
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only  for  the  instance  of  this  research  and  may  not  be 
appropriate  in  other  circumstances,  particularly  if  trying 
to  design  an  actual  decision  support  system  for  use  in  a 
source  selection  activity.  dBASE  II  has  major  limitations 
which  would  indicate  its  unsuitability  for  many  operational 
applications.  For  example,  the  limitations  on  record  size 
(ie.  the  number  of  characters  per  record  is  1,000  maximum, 
fields  per  record  32  maximum,  and  characters  per  field  254 
maximum),  the  slow  speed  of  sorting  oprations,  and  its 
limited  search  capabilities  would  probably  severely  restrict 
its  use  in  a  DSS.  Indeed,  the  restriction  of  a  maximum  of 
32  fields  per  record  slightly  degraded  its  effectiveness  in 
this  application  and  will  lead  to  a  recommendation  in 
Section  V  that  future  researchers  should  move  to  a  system 
with  greater  capacity. 

Data  Processing 

The  data  was  entered  into  the  dBASE  II  database  as 
closely  as  possible  to  verbatim  from  the  set  of  Source 
Selection  Plans.  In  most  plans  the  source  selection 
criteria  areas  and  items  were  given  as  paragraph  headings 
and  direct  transcription  was  possible.  A  few  plans  however 
included  the  description  of  the  criteria  within  prose  and 
the  author's  best  interpretation  of  the  criteria  was 
recorded.  Wherever  possible  key  words  were  transcribed  from 
within  descriptive  paragraphs. 
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Each  record  within  the  •database;  then  consisted  of  32 
fields  containing  the  f o  1  lowing  info r  ma  t  i  oh:  i«.  ^  ~ 


1. 

Name 

of  project  ■•**?’■* 

2. 

Phase  of  the  Acquisition  cycle 

3. 

General  description-  •  *•'  -  ■  '*'  ~ 

4. 

General  description 

5. 

General  'description  ‘  -•  —  • 

6. 

Contract  type 

7* 

Approximate  date*  6 f  the*  Source  Select  ion1 

8. 

Area 

9i 

Item  : *>■'  ■;  ^ 

10. 

Item 

11. 

Item  ■*  fV  f  ‘  *•  - 

12. 

Item 

13. 

Area 

•' . ' :  '  /'  • 

14. 

Item 

15. 

Item  '•*  -•  *  1  '■'*  <*■' 

16. 

Item 

17. 

Item  ’ 

18. 

Area 

>  *  . 

19. 

- 

'Item'--  *  -  ">*  •  i  ••  ‘ 

20. 

Item 

21. 

Item  '  •  "  'f '  ** 

22. 

Item 

23. 

Area 

'  .  i  ■  ;  ;  '  1  •*  . 

24. 

Item 

25. 

Item 

26. 

Item 

27. 

Item 

28. 

Area 

29. 

Item 

30. 

Item 

31. 

Item 

32. 

Item 

The 

name 

of  the  project  was  included  for  each  source 

selection  plan  to  facilitate  reference  to  the  original  data 
by  the  author  for  data  entry  confirmation  and  review,  but 
due  to  the  sensitive  nature  of  source  selection  information 
and  the  undertaking  given  by  the  author  to  the  data  source 
at  Aeronautical  Systems  Division,  the  project  names  will  not 
be  included  in  any  output  published  in  this  document. 
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The  following  five,  data  fields  provide  a  keyword 
description  of  the  weapon  system  being  procured  including, 
in  the  first  field,  the  phase  of  the  acquisition  cycle 
during  which  this  source  selection  was  initiated,  and  in  the 
last  field  the  type  of  contract  to  be  awarded,  if  known. 
These  data  fields  should,  collectively,  and  in  broad  terms 
describe  the  type  of  system  and  the  type  of  procurement 
activity  as  it  may  have  relevance  to  the  determination  of 
the  source  selection  criteria. 

The  date  field  was  included  only  to  facilitate 
sorting  the  data  into  rough  chronological  order  and,  for  the 
reasons  of  source  selection  information  sensitivity  cited 
previously,  will  not  be  output  in  any  reports.  The 
desireability  of  having  the  data  arranged  in  rough 
chronological  order  lies  in  the  assumption  that  if  program 
managers  do  indeed  refer  to  recent  previous  source 
selections  for  guidance  in  the  determination  of  source 
selection  criteria,  patterns  of  similarity  between  criteria 
sets  may  be  more  easily  recognised  during  a  scan  of  the  data 
if  the  plans  are  grouped  according  to  temporal  criteria. 

Data  fields  were  then  reserved  for  entry  of 
evaluation  criteria  including  up  to  five  areas  with  up  to 
four  items  per  area.  This  was  the  most  significant 
limitation  imposed  on  the  research  by  the  selection  of  dBASE 
II  as  the  manipulation  tool.  Only  a  total  of  25  data  fields 
were  available  for  entry  of  the  evaluation  criteria  areas 


and  itemsi  .  While  the  five  area  fields  were  adequate  for  all 
the  Source  Selection  Plans  in  the, data  set,  several  planSr 
contained  more  than  four  items  in  one  or  more  areas  and 
very  few  plans  contained  a  large  number  of^items  in  one  or 
two  areas^  However,  in  these  cases  the  items  were  very 
system  specific,  were  not  matched  by  any  other  plan  in  the 
data  evaluation  criteria  for  a  very  closely  related  system. 
Thus  their  necessary  ommission  from  the  database  was  not 
considered  a  significant  cost  in,  Shis  effort  when  related  to 
the  other  benefits  of  using  the  d BASE  II  program.  In  other 
situations  this  may  not  be  the  case  and  the  use  of  an 
alternative  database  management  system  may  be  desirable. 

The  database  was  sorted  on  the  date  field  into 
ascending  order  and  all  fields  except  the  project  name  and 
date  were  printed  out. 

■  ,  •  •  it  *  '  ■*»  f  r' 

During  entry  of  the  data  into  the  database  the 
observation  was  made  that,  of  the  large  variety  of 
evaluation  criteria  areas  and  items,  many  were  substantially 
equivalent,  differing  only  in  nomenclature.  In  order  to 
rationalize  the  data  the  printed  output  of  the  original  data 
base  was  examined,  synonomous  areas  and  items  were 
identified,  and  the  most  frequently  used  term  for  that  area 
or  item  was  added  to  a  parsimonious  set  of  area  and  item 
names.  A  copy  of  the  original  database  file  was  made  and 
this  copy  was  edited  by  substituting  synonomous  area  and 
item  names  in  conformity  with  the  parsimonious  set  of 
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criteria.  A  tabulation  of  the  original  areas,  their 
frequency  of  use,  and  the  substitutions  used  is  presented  in 
appendix  A. 

Proceeding  with  the  revised  dataset  the  following 
statistics  were  collected: 

a.  Use  of  each  area.  The  total  number  of 
instances  of  the  use  of  each  area  in  the  sample 
and  percentage  proportion  of  usage  in  the 
sample  were  recorded. 

b.  Use  of  each  Item.  The  total  number  of 
instances  and  percentage  uses  of  each  item  in 
the  sample  and  also  within  each  area  were 
recorded. 

These  statistics  were  used  to  create  a  further  dBASE 
•  II  database  from  which  were  listed  the  areas  and  items  in 
relative  order  of  usage  frequency,  in  order  to  observe  the 
most  popular  areas  and  items  and  the  relative  strengths  of 
their  popularity.  The  result  of  this  processing  provides  a 
hierarchical  list  of  areas  (tabulated  in  Appendix  B),  and 
their  relative  items  (Appendix  C),  indicating  the 
historically  based  relevance  of  those  areas  and  items  to  the 
source  selections  of  the  population  sample.  Therefore,  if 
the  assumption  that  past  experience  in  the  selection  of 
source  selection  evaluation  criteria  is  relevant  to  the 
determination  of  criteria  for  new  programs,  the  list  also 
provides  a  prioritized  clue  to  the  criteria  which  should  be 
examined  for  inclusion  in  any  new  program. 


In  order  to  determine  the  frequency  of  occurrence  of 
each  area,  the  respective  area  fields  of  each  record  in  the 
original  database  were  sorted  into  alphabetical  order  and 
output  in  vertical  columns  placed  side-by-side  across  the 
printer  page.  This  had  the  advantage  of  not  only  arranging 
the  areas  into  easily  identified  groups  with  the  relative 
frequency  of  the  most  common  area  names  easily  observable, 
but  also  clearly  showed  the  numbers  of  source  selections 
that  used  only  two  areas,  three  areas,  four  areas,  and  five 
or  more  areas  in  the  evaluation  criteria. 

The  structured  listing,  while  presenting  the  program 
manager  with  a  comprehensive  selection  of  possible  criteria 
together  with  their  relative  probability  (historically 
speaking)  of  being  useful  to  him,  is  not  a  particularly 
useful  decision  support  aid  as  it  stands  because  it  simply 
presents  a  large  volume  of  information  in  an  ordered  form 
rather  than  presenting  selective  information  tailored  to  his 
particular  requirements.  The  decision  maker  needs  to  be 
able  to  call  only  that  information  which  is  clearly  relevant 
and  important  to  the  decision  at  hand  and  discard  and 
suppress  any  superfluous  or  trivial  information  which  may 
serve  to  cloud  the  issue  rather  than  support  a  decision. 

The  process  of  synthesis  of  the  decision-supporting 
information  requires  an  input  from  the  decision  maker  to 
define  the  decision  parameters  and  thus  give  the  decision 
support  system  direction.  The  input  which  the  program 
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manager  has  to  support  his  source  selection  evaluation 
criteria  decision  was  described  earlier  (Figures  3  and  4). 

The  original  dataset  included  five  fields;  per  record 
refered  to  as  attributes  of  the  system  which  was  the  subject 
of  the  source  selection.  The  attributes  of  the  system 
serve  to  record  a  keyword  description  of  the  type  of 
procurement  activity,  including  the  phase  of  the  program, 
the  type  of  equipment  or  service,  and  the  type  of  contract. 

The  aggregate  of  the  attributes'  of  all  the  systems 
in  the  dataset  may  be  considered  to  be  members  of  a  set. 
Furthermore,  if  the  membership  of  the  set  is  sufficiently 
comprehensive,  any  new  system  will  very  likely  be 
describable  in  terms  of  a  selection  of  some  members  of  the 
set,  a  specific  subset  of  attributes. 

A  subset  of  this  set  would  be  associated  with  each 
system  in  the  dataset,  and  each  such  subset  would  also  be 
associated  with  a  subset  of  the  set  of  source  selection 
evaluation  criteria.  Considering  the  dataset  as  a 'whole, 
the  attributes  could  be  regarded  as  independent  variables 
and  the  respective  evaluation  criteria  as  dependent 
variables.  If  a  causal  relationship  between  the  independent 
and  dependent  variables  could  be  v hown  to  exist,  then  there 
would  be  a  basis  for  the  postulation  that  given  any  subset 
of  attributes  a  set  of  evaluation  criteria  could  be 
predicted.  In  other  words,  if  a  new  system  can  be  described 
with  a  subset  of  the  set  of  attributes  derived  from  the 
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historical  <Uta,  then  a  set  of  probably  appropriate 

'i  ‘  '  *  *  V‘  *  ' '  '  *  -  -y  v  •  '  '  *C  ,  i'*V  *  7.  <  J  M  $ 

evaluation  criteria  could  be  determined  based  solely  on  the 
historic  data.  This  would  provide  the  core  of  a  worthwhile 

njv  «  '  V„‘  ‘  .  ■' '  ’  -‘j”  '  -f  ';r-.  J.  ’  "s.  '  " 

decision  support.  tool. .  ,  ,  .  ...  , \ .  ,  ,  „  .  >!t  _  A.t 

Statistical  techniques  are  available  which  can 

■  -  >  2.  'v  '*  -  * v  >  *■»  •  -  *  -  ^  r-  J;  *' •"  ''i,,vr  »  ..  ~  *«v  ,\j  •  y 

assess  the  probability  of  a  causal  r el a ti o n ship  existing 
between  two  or  more  nominal  variables.  These. techniques 


fall  into  the  realm  of  analysis  of  cross-classified 
categorical  data  and  are  well  described?  liy-va^otts  texts  . 
including  Siegel  (32),  Reynolds  (30),  and  Fiehberg  (19). 
Briefly,  the  procedure  consists  of  thecdnstructing  a 
contingency  table  in  two  or  more  {dimensions  and  using 
various  statistical  tests,  such  as  chi-square  of  the 
contingency  co-efficient,  to  assess  the  degree  of 

f  .  +  '  .  V  .  .  - 

association  between  the  variables. 

However,  these  techniques  only  allow  for  the 
possibility  of  one  value  for  each  variable  to  occur  in  any 
one  observation.  That  is,  for  any  observation  in  the  data  , 
set,  the  possible  values  for  any  variable  must  be  mutually 
exclusive.  The  data  set  constructed  for  this  thesis  does 
not  comply  with  those  constraints  if  the  system  is  taken  as 
a  variable  and  the  system  attributes  as  values  of  that 
variable.  The  variable  "System”  will  have  several  values 
simultaneously  in  each  observation  of  the  dataset.  A 
methodology  to  determine  the  association  or  otherwise 
between  the  values  of  the  variable  "System"  and  the 
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evaluation  criteria  by  considering  all  attributes  of  the 
system  together  would  be  desirable  to  take' "  account  of  any 
modifying  influences  between  the  attributes  in  order  to 
moderate  the  indication  of  false  associations  occurring  as  a 
result  of  relationships  between  the  values  of  the  variable 
"System”.  A  search  of  the  literature  has  failed  to  reveal 
any  methodology  which  specifically  approaches  this  problem^ 

Heuristic  Methodology 

There  is,  however,  the  possibility  of  constructing  a 
less  rigorous  heuristic  methodology  for  analysis  of  the  data 
and  subsequent  decision  support..  The  .method  of  description 
of  each  system  must  be  modified  to  permit  adequate 
discrimination  of  a  system  by  selecting  a  value  for  each 

\  ’  h' 

variable  in  a  set  of  variables  which  collectively  describe 
the  system.  Each  variable  should  have  a  limited,  and 
mutually  exclusive  set  of  values  from  which  one  must  be 
selected  for  each  observation  in  the  data  set. 

To  illustrate  the  procedure  consider  five  possible 
variables  which  may  be  used  to  describe  a  system.  They  are: 

a.  The  phase  of  the  acquisition  cycle,  having  the 

values; 

(i)  conceptual, 

(ii)  demonstration  and  validation, 

(iii)  full  scale  development,  or 

(iv)  production  and  deployment. 
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b.  Major  system,  a  dichotomous  variable  with 
values  yes  or  no  determined  in  accordance  with 
AFR  70-15  or  other  approprav.  authority. 

c.  The  class  of  weapon  system  with  values,  for 
example: 

(i)  aircraft, 

(ii)  missile, 

(iii)  vehicle, 

(iv)  ground  radar,  etc* 

d.  The  scope  of  the  procurement  with  values,  for 
example: 

(i)  complete  system, 

(ii)  spare  part, 

(iii)  component  breakout, 

(iv)  integrated  subsystem  (eg.  radio,  ECM, 
navigational  aid,  software), 

(v)  ground  support  equipment, 

(vi)  maintenance  services,  etc. 

e.  The  class  of  contract  to  be  awarded,  for 
example; 

(i)  firm  fixed  price  (FFP), 

(ii)  cost  price  incentive  firm  (CPIF), 

(iii)  co&t  price  award  fee  (CPAF),  etc. 
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The  actual  variables  and  their  respective  value 
sets  used  in  any  specific  decision  support  system  would  need 
to  be  selected  following  careful  assessment  of  the  most 
important  factors  bearing  on  the  selection  of  evaluation 
criteria  for  the  type  of  systems  normally  procured  by  the 
particular  agency  employing  the  DSS.  The  preceding 
variables  are  presented  purely  as  a  illustration  of  the 
procedure  and  in  no  way  suggested  as  being  appropriate  for 
any  instance. 

Having  assigned  an  appropriate  value  to  each  System 
variable  for  each  observation  in  the  dataset,  one  may  now 
construct  a  series  of  conventional  contingency  tables  for 
each  system  variable  against  a  variable  (say  Areas)  which 
has  as  its  values  the  evaluation  criteria  areas  for  the 
observations  in  the  data  set.  If  each  observation  has  five 
criteria  areas  (including  possibly  one  or  more  null  entries, 
that  is,  some  observations  may  have  for  example  only  three 
evaluation  areas  and  thus  would  have  two  null  or  "No 
Criteria"  entries)  then  the  totals  of  the  rows  and  columns 
of  the  contingency  table  will  be  greater  by  a  factor  of  five 
than  would  be  the  case  if  there  was  only  one  value  of  the 
variable  "Areas"  per  observation  in  the  dataset.  The  cells 
of  the  contingency  table  could  therefore  be  normalized  by 
dividing  throughout  by  five  and  then  percentages  calculated 
for  each  cell  in  the  normal  way. 
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A  hypothetical  construction  of  such  a  table  is  given 
as  Table  I.  In  this  case  the  system  variable  is  the  "Phase 
of  the  Acquisition  Cycle"  with  its  five  possible  values  as 
illustrated  earlier.  The  other  variable  is  "Areas"  with 
seven  possible  values; 

a.  Technical, 

b.  Operations, 

c.  Management, 

d*  Production, 

e.  Logistics, 

f.  Cost,  and 

g.  No  Criteria. 

Assume  that  this  hypothetical  dataset  contains  100 
observations  (that  is,  100  different  systems  which  underwent 
the  source  selection  process)  and  each  observation  contained 
exactly  five  evaluation  criteria  areas  (of  which  one  or  more 
may  have  been  null  or  no  criteria  where  the  system  was 
evaluated  using  less  than  five  evaluation  criteria  areas). 
Into  each  cell  of  the  table  is  entered  the  number  of  records 
of  the  data  set  for  which  the  relevant  values  of  each 
variable  co-exist.  For  example,  there  may  be  22  systems  out 
of  the  total  of  100  which  were  in  the  "conceptual"  phase  of 
the  acquisition  cycle,"  and  of  those  22  they  all  used  the 
criteria  "Technical"  and  "Cost,"  only  five  used 
"Operations,"  20  used  "Management,"  two  used  "Logistics," 
and  none  used  "Production."  It  follows  then  that  there  were 
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TABLE  I 


Example  Contingency  Table 


Variable:  Acquisition  Cycle  Phase 
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25 

33 
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39 
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Bn 
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mi 
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(22.0) 
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100.0% 

100.0% 
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100.0% 
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39  blank  "areas”  fields  within  those  22  observations  of  the 
data  set  and  they  are  recorded  in  the  "no  criteria"  cell. 
When  the  counts  for  each  cell  are  entered  the  table  may  be 
normalized  by  dividing  throughout  by  five  because  there  were 
five  values  for  "Areas"  recorded  in  each  observation  of  the 
data  set.  Now  the  total  sum  of  all  the  cells  of  the  table 
is  100  which  equates  to  the  number  of  records  in  the  data 
set  as  is  normal  for  a  standard  two-dimensional  contingency 
table.  The  values  in  the  table  may  also  be  expressed  as 
percentages  by  dividing  throughout  by  the  number  of  records 
in  the  data  set  if  that  is  the  preference  of  the  user. 

Precisely  the  same  procedure  for  developing  a 
contingency  table  could  be  followed  for  each  of  the  other 
"System"  variables,  resulting  in  a  set  of  contingency 
tables.  Each  of  the  contingency  tables  may  then  be  examined 
for  statistically  significant  association  between  the  system 
variable  and  the  evaluation  criteria  areas  used.  The 
statistical  analysis  could  be  very  easily  accomplished  using 
standard  analysis  techniques  such  as  those  described  by 
Fienlerg  (19),  Siegel  (32),  or  any  other  text  on  the 
analysis  of  cross-classified  categorical  data  and  with  the 
aid  of  one  of  the  several  statistical  software  packages  such 
as  SPSS  (Statistical  Package  for  the  Social  Sciences)  that 
are  commercially  available. 

The  aim  of  the  statistical  analysis  should  be  to 
determine  whether  there  is  any  association  between  the 
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evaluation  criteria  and  each  system  variables  If  there  is, 
then  the  system  variable  may  be  included  in  the  model  for 
decision  support,  but  if  there  is  not  determined  to  be 
statistically  significant  association  for  any  particular 
system  variable,  then  that  variable  should  probably  be 
discarded  as  not  being  a  useful  description  of  a  system. 

The  choice  of  one  or  several  of  the  measures  of  association- 
applicable,  to  multivariate  tables  of  nominal  data  is- a 
matter  for  the  personal  preference  of  the  user  and  will  not, 
be  explored  here. 

Consider  now  the  case  of  a  new  system  which  is  being 
procured  and  subject  to  source  selection  procedures.  This 
new  system,  in  exactly  the  same  manner  as  each  of  the 
systems  in  the  data  set,  could  be  described  by  selecting  a 
value  for  each  of  the  previously  defined  system  variables. 
These  values  may  be  used  as  labels  for  columns  in  a  table 
such  as  presented  in  Table  II.  The  rows  of  the  table  may  be 
labelled  with  the  possible  values  of  the  variable  "areas"  in 
identical  manner  to  the  set  of  contingency  tables.  Into  the 
cells  of  each  column  of  this  table  may  be  transcribed  the 
values  from  the  cells  of  the  column  of  the  same  name  in  the 
appropriate  contingency  table.  We  now  have  a  table  which 
gives  an  indication  of  the  relative  frequency  of  use  of 
criteria  in  previous  source  selections  for  systems  with  one 
of  the  characteristics  of  the  new  system.  An  aggregation  of 
these  historical  associations  should  provide  an  indication. 
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:  TABLE  II 

Example  Decision  Support  Prediction  Table, 
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System 

Phase 
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System 
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• 
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22 
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suitable  for  implementation  within  a  decision  support 
system,  of  which  criteria  are  likely  to  be  relevant  in  the 
source  selection  process  for  a  new  system  with  these 
characteristics.  The  simple  summation  of  the  cells  in  each 
row  will  give  figures  for  each  criteria  which  reflect  the 
frequency  of  use  of  that  criteria  relative  to  the  other 
possible  criteria  in  the  historical  data.  Thus  the  criteria 
may  be  ranked  in  order  of  past  usage  and  indicate  a 
priority  for  consideration  by  the  program  manager  for  use 
with  his  new  system.  Criteria  with  very  low  scores  may  even 
be  excluded  from  presentation  by  the  decision  support  system 
for  consideration  by  the  program  manager.  The  score  of  the 
"no-criteria"  row  indicates  the  degree  to  which  the  past 
source  selections  used  less  than  five  areas  in  their 
criteria  and  thus  a  high  score  would  signify  to  the  program 
manager  that  he  probably  should  also  consider  selecting 
fewer  criteria  for  the  new  system  evaluation. 

Assumptions 

The  procedure  outlined  is  based  on  some  rather  broad 
assumptions  as  is  the  case  with  many  useful  heuristic  tools. 
However,  the  limitations  imposed  by  these  assumptions  may  be 
reduced  by  careful  design  of  the  DSS  and  by  user  awareness 
of  the  assumptions  and  recognition  of  the  role  of  a  DSS  as  a 
decision  support  tool  rather  than  a  decision  making  system. 

The  first  and  major  assumption  is  that  of 
independence  between  the  various  descriptive  system 
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variables.  For  a  data  set  gathered  to  provide  the  basis  for 
a  specific  DSS  implementation,  the  systems  analyst  could 
test  the  system  variables  proposed  for  use  and  hopefully 
only  incorporate  those  which  were  both  significant  to  the 
selection  of  evaluation  criteria  and  yet  not  themselves 
significantly  associated. 

Success  in  the  selection  of  mutually  independent 
system  variables  will  reduce  the  limitations  of  the  second 
assumption  that  spurious  associations  indicated  between  the 

variables  in  each  contingency  table  are  not  significant. 

* 

Prudent  selection  of  the  measures  of  association  by  the 
analyst  implementing  the  DSS  will  reduce  this  danger. 

A  third  assumption  is  that  the  data  set  reflects 
successful  source  selection  processes  and  some  measure  of 
the  eligibility  of  a  particular  source  selection  for 
inclusion  in  the  data  set  would  be  desirable.  This  may,  in 
practice,  be  an  extremely  difficult  factor  to  evaluate  but 
its  importance  is  obvious.  Unless  some  sort  of  feedback 
mechanism  was  built  into  the  DSS  to  record  the  success  or 
otherwise  of  the  decisions  made,  the  quality  of  decisions 
will  tend  toward  a  mean  but  not  improve  although,  of  course, 
they  may  be  made  more  quickly. 

Limitations  on  the  Research 

The  author  was  unfortunately  not  able  to  pursue  the 
validation  of  the  heuristic  methodology  to  the  degree  that 
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he  desired.  The  prime  reason  was  that  he  did  not  have 
access  to  the  level  of  information  required  to  assign  values 
to  the  system  variables  for  the  dataset.  To  proceed  further 
with  a  relatively  arbitrary  assignment  of  values,  based  on 
inadequate  information,  and  then  draw  conclusions  from  the 
results  would  be  simply  futile  and  probably  misleading.  The 
required  level  of  access  to  the  necessary  source  selection 
information  could  probably  only  come  in  conjunction  with  a 
formal  research  directive  from  an  appropriately  high 
management  level  within  the  relevant  Command. 
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$ V .  Findings 


The  Problem 

The  Source  Selection  Process  is  almost  invariably  a 
complex,  time  consuming,  and  costly  process.  The  high  cost 

4  *  ’  *  *  r  ’  v  r  t  ,  .  » '  ,  1 

is  largely  a  function  of  time  and  thus  one  method  of 
significantly  reducing  costs  is  to.  streamline  the  process. 

An  avenue  being  actively  pursued  is  computerization  of  some 
areas  of  source  selection  activities,  notably  office 
automation,  since  a  large  volume  of  work  is  typically, 
clerical  in  nature  (5,  6,  17),  and  program  managers’ 

i  i  *’ }  •  x  '  » 

decision  support  since  many  of  the  decisions  are  complex 
(12,  13,  31).  The  aim  of  this  research  effort  was  to 
further  aid  in  the  provision  of  decision  support  for  program 
managers  during  the  determination  of  proposal  evaluation 
criteria  for  source  selection. 


The  P>esearch  Question 

Given  that  the  selection  of  appropriate  evaluation 
criteria  in  any  source  selection  is  not  only  difficult  and 
time  consuming,  but  also  crucial  to  the  success  of  the 
project,  the  question  is  whether  a  decision  support  system 
designed  to  aid  the  program  manager  in  this  task  is 
feasible. 
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The  Data 


The  database  used^for; jthe  .research  was  drawn  from 

the  Source  Selection  Plans  of  81  source  selections  processed 

l.-  }y-.i  ' 

by  Air  Force  Systems  Command,  Aeronautical  Systems  Division 
(ASD)  since  1971.  Restrictions  imposed  by  the  Source 
Selection  Sensitive  classification  of  the  data  were  non¬ 
disclosure  of  sensitive  information  in  this  report,  and 

‘  %  f  ■*  jr »  -X-'  <5  *V  -  *-  1  1  *3  I 
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denial  of  access  by  the  author  to  further  sensitive  system/ 
information.  Thus  the  research  waS  halted  at  the  point 
where  a  higher  level  of  security  clearance  was  required  to 
proceed. 


The  data  was  intended  to  serve  only  aS  a  base  from 

•  ,  <  ■  *,  ' 
which  to  determine  the  feasibility  of  the  ideas  to  be 

introduced,  not  as  an  information  source  for  an  operational 

decision  support  system.  As  such,  it  needed  only  to  be 

representative  of  real  world  data,  not  a  total  population  or 

even  a  strictly  random  sampling. 


The  Database  Management  System 

Manipulation  of  the  data  was  achieved  using  the 
relatively  simple,  convenient,  and  economical  relational 
database  management  system  for  microcomputers  called  dBASE 
II  (copyright  by  Ashton  Tate,  Culver  City  CA).  The  system 
proved  to  have  significant  limitations,  principally  in  terms 
of  capacity  and  speed  of  processing,  which  would  render  it 
unlikely  to  be  suitable  for  use  in  an  operational  DSS,  but 
for  the  purpose  of  examining  this  data  set  it  was  adequate. 
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Observations  from  the  Data 

*“  '  '  £'■  v/VA  \  l :: 

Several  observations  were  made  from  the-  data  during 


manual  entry  into  the  database  and  prior  to  any  processing. 
The  observations  related  to  the  general  form  of  evaluation 
criteria. 

>  1  .  **  *'  *  t  4k  ‘  * t  c*  k:  ”  V  ^  S  *  v* : 

Fields  were  reserved  in  the  database  file  to  record 

^  *  v  ^  y'/  ■*  ^ 1  *  j  -  '  ^  ^  /  tt 

five  criteria  areas  per  record  (a  source  selection  plan  or 
system  undergoing  source  selection)  and  four  criteria  items 


per  area.  The  space  reserved  for  areas  proved,  to  be 
adequate  in  all  except  a  very  few  cases  and  those  plans 
which  used  more  than  five  evaluation  areas  either  spilt  an 
otherwise  common  area  into  two  or  added  an  area  or  two  which 
were  unique  to  that  particular  system  and  thus  would  likely 
have  significance  in  a  DSS  which  was  concerned  with 
highlighting  frequency  of  useage.  The  space  reserved  for 
items,  limited  by  the  dBASEII  capacity  was  often 
insufficient  and  thus  detailed  analysis  of  items  was  not 
possible. 

A  second  observation  was  that  there  was  considerable 
variety  in  the  method  of  presenting  the  areas  and  items  in 
the  plans.  Many  plana  simply  listed  them  or  presented  them 
as  paragraph  or  sub-paragraph  headings  making  up  a  form  of 
qualified  and  amplified  listing.  A  few  plans  included  the 
presentation  of  the  criteria  in  a  narrative  form  where  the 
author  was  compelled  to  extract  key  words  from  a  paragraph 


or  try  to  capture  the  essence  of  the  criteria  as  described 

'  4  .  -  ' 

with  one  of  the  more  commonly  used  terms.  This  was  a  rather 
subjective  process,  but  with  little  alternative; 

A  further  observation  was  that  many  of  the  criteria, 
particularly  areas,  used  a  variety  of  terms  to  refer  to 
essentially  the  same  thing.  For  example,  "manufacturing” 
and  "production"  were  both  used  but  the  author  could  not, 
with  the  information  available  to  him,  determine  any 
difference  in  meaning  between  the  two.  As  will  be  discussed 
later,  the  rationalization  of  all  synonomous  terms  into  a 

.  'u  'l  .  j,  » 

parsimonious  set  of  criteria  names  produced  a  very  much 
reduced  number  of  different  criteria  which,  had  they  been 
used  in  the  original  plan,  would  have  produced  much  more 
marked  similarity  between  the  plans. 

Even  without  the  rationalization  of  the  area  names 
there  was  observed  to  be  regular  similarities  among  the 
plans,  particularly  when  arranged  in  rough  chronological 
order.  The  similarities  suggest  two  likely  possibilities. 
Either  reference  is  commonly  made  to  the  files  of  previous 
source  selections  during  the  determination  of  source 
selection  criteria  as  was  discussed  on  pages  37  and  39  and 
corroborated  by  a  professional  in  the  field,  or  plans  are 
commonly  prepared  by  reference  to  a  standard  handbook, 
model,  or  authority.  Whichever  is  the  case,  a  successful 
DSS  could  supercede  both  procedures. 
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Following  entry  of  the  date  into  the  database  some 
summary  statistics  were  able  to  be  gathered.  The  database 
area  fields  were  sorted  into  alphabetica1  order  and  printed 
cut  in  vertical  columns  placed  side  by  side  across  a  wide 
page.  From  this  printout  a  simple  count  showed  that  64  (or 
79%)  of  the  81  plans  in  the  dataset  used  less  than  five 
criteria  areas  and  of  those,  24  (30%  of  the  dataset)  used 
less  than  four  areas.  Consequently,  the  assumption  that 
five  criteria  areas  are  normally  adequate  to  cover  the 
requirements  of  any  source  selection  situation  appears 
reasonably  valid. 

The  data  were  then  examined  for  the  frequency  of 
occurrence  of  each  criteria  area.  There  were  73  different 
areas  used  in  the  set  of  81  plans.  Of  those  73  the  most 
commonly  used  criteria  areas  were: 


Area  Occurrence  in  81  Systems  % 


Cost/Price 

60 

74 

Technical 

53 

65 

Management 

41 

51 

Logistics 

20 

25 

Life  Cycle  Cost 

15 

19 

Operational 

8 

10 

The  remaining  areas  were  all  used  in  less  than  10% 
of  instances  and  forty  seven  areas  were  used  only  once. 
However,  as  stated  previously,  many  of  the  areas  used 
infrequently  appeared,  given  the  limited  information 
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available  to  the  author,  synonomous  with  one.  of  the  more 
popular  terms.  Further,  some  areas  were  really  the 
conjunction  of  two  areas,  as  in  the  cases  for  example  of 
"Operational/Technical,"  '’Logistics/Operations, n  and 
'’Technical/Operational  Utility." 

A  very  few  areas  did  not  easily  fall  into  any  common 
category  and  whether  or  not  their  importance  was  sufficient 
in  the  source  selection  for  which  they  were  used,  to  warrant 
inclusion  as  an  exclusive  area  could  not  be  satisfactorily 
determined  by  the  author  from  available  information.  The 
criteria  "Weight"  and  "Test-in-Container  Capability"  were 
examples  of  unique  criteria  areas  in  the  dataset  which  the 
author  questions  may  have  more  appropriately  been  evaluated 
as  highly  weighted  items  under  the  area  of  "Operations." 

But  supportable  judgement  on  these  matters  could  only  be 
made  with  the  benefit  of  an  indepth  knowledge  of  the  system. 

The  approach  then  taken  was  to  examine  the  data  in 
relation  to  those  criteria  which  are  prescribed  or  mandatory 
according  to  Federal  or  Defense  regulations. 

The  Federal  Acquisition  Regulations  (41:15-20) 
specifically  states  that  "price  or  cost  to  the  Government 
shall  be  included  as  an  evaluation  factor  in  every  source 
selection."  There  is  thus  no  doubt  that  "Cost/Price"  should 
be  included  in  the  set  of  mandatory  evaluation  criteria. 

The  source  selections  in  the  dataset  largely  complied  with 
that  requirement  although  a  significant  number  (19%) 


72 


included  the  cost/price  evaluation  under  the  area  name  of 
wLife  Cycle  Cost.” 

The  only  other  criteria  that  was  noted  to  be 

specifically  prescribed  for  use  was  that  of  "Contractor  Past 

Performance"  as  directed  by  both  DODD  5000.34,  Defense 

Production  Management  (37)  which  states; 

Contractor  past  performance  (to  the  extent  that  it  has 
a  bearing  on  the  concept  involved),  potential  to 
execute  the  production  program*  and  demonstrated 
production  management  capability  shall  be  among  those 
factors  .  .  .  evaluated  ...  in  the  source  selection 

and  Interim  Message  Change  (IMC)  79-1  to  AFR  70-15/AFSC 

Supplement  1  which  implemented  the  use  of  Relevant  Past 

Performance  in  all  formal  source  selections. 

However,  the  interpretation  applied  to  that  directive 

by  ASDP  800-7  (43:5)  is  that  past  performance  in  this 

instance  is  considered  a  type  of  evaluation  measurement  such 

as  understanding  the  problem,  soundness  of  approach, 

compliance  with  requirements  correction  potential  (impact  on 

design),  effect  on  schedule,  and  unique  solutions.  That 

position  is  further  reinforced  by  the  latest  publication 

concerning  source  selection,  AFR  70-15,  issued  on  22 

February  1984.  Specifically,  it  states  that  past 

performance  may  be  both  an  assessment  criterion  and  a 

general  consideration,  but  does  not  suggest  that  it  should 

be  considered  as  a  specific  criterion. 

Thus  one  would  not  expect  to  see  "Contractor  Past 
Performance"  as  an  evaluation  area,  but  as  a  measure  of 
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evaluation  for  all  areas.  This  is  supported  by  observation 
of  the  dataset  since  only  7%  of  the  source  selections  used 
"Contractor  Past  Performance”  as  an  area,  but  most  mentioned 
it  as  a  factor  in  the  explanatory  narrative  of  the  source 
selection  plan.  One  must  also  note  that  the  entire  dataset 
is  composed  of  plans  which  were  formulated  prior  to  the 
publication  of  the  more  explicit  guidance  contained  in  the 
current  issue  of  AFR  70-15.  In  view  of  the  very  recent 
guidance  contained  in  AFR  70-15,  Past  Performance  does  not 
now  strictly  qualify  for  inclusion  in  any  set  of  specific 
criteria. 


The  review  of  the  regulatory  publications  revealed 
that  certain  evaluation  areas  take  on  the  qualification  of 
recommended  criteria  simply  because  of  their  repeated  use  as 
examples  of  appropriate  criteria  in  numerous  publications  at 
all  levels.  Those  criteria  areas  are: 

a.  Technical, 

b.  Operations, 

c.  Logistics, 

d.  Management, 

e.  Production,  and 

f*  Costs  * 

The  data  generally  supports  the  view  that  these  six 
areas  should  be  included  in  a  set  of  recommended  criteria. 
They  were  the  most  frequently  used  criteria  with  the 
exception  that  Life  Cycle  Cost  was  more  common  than 
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Operations.  The  author  feels  that  life  cycle  cost  could 
reasonably  be  considered  a  sub-part  of  the  more  generic  term 
Costs.  Then  the  Cost/Price  area  would  have,  with  the 
substitution  of  cost  for  life  cycle  cost,  tallied  75  out  of 
81  systems  or  82%  of  the  source  selections  in  the  dataset. 

In  summary,  the  only  compulsory  area  is  Cost/Price 
and  the  five  next  most  common  and  therefore  recommended 
criteria  areas  are  Technical,  Operations,  Logistics, 
Management,  and  Production.  The  remainder  of  the  criteria 
in  the  dataset  would  then  be  relegated  to  the  category  of 
system  specific  criteria. 

A  closer  examination  of  the  dataset  however  shows 
that  the  majority  of  the  now  system  specific  criteria  are 
very  closely  related  to,  subparts  of,  or  synonomous  with, 
one  or  other  of  the  compulsory  or  recommended  criteria.  In 
order  to  rationalize  the  criteria  sets  the  system  specific 
criteria  in  the  dataset  were  replaced  with  the  appropriate, 
more  generic  term  in  the  compulsory  and  recommended  sets. 
This  substitution  was  a  somewhat  arbitrary  process  but  done 
with  some  care  by  the  author  and  the  substitutions  are 
listed  in  Appendix  A  for  the  reader  to  make  his  own 
assessment. 

Analysis  of  the  new  dataset  will  not,  of  course, 
change  the  membership  of  the  set  of  compulsory  criteria 
since  that  was  determined  solely  by  reference  to  regulation. 
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The  incidence  of  use  of  each  of  the  criteria  areas 


in  the  new  dataset  is  as  follows  (also  repeated  at 
Appendix  B): 


AREA 

INCIDENCE 

%  USAGE 

Contractor  Capability 

6 

7 

Contractor  Past  Performance 

6 

7 

Coproduction/Offset 

2 

2 

Cost/Price 

75 

92 

Logistics 

40 

49 

Logistics/Management 

1 

1 

Logistics/Operations 

1 

1 

Management 

57 

70 

Management/Production 

8 

9 

Operations 

24 

29 

Production 

12 

14 

Schedule 

8 

9 

Technical 

66 

81 

Technical/Operations 

6 

7 

Technical/Production 

1 

1 

Interestingly,  the  order  of 

descent  from 

most 

commonly  used  criteria  to  least  commonly  used  has  not 

changed  for  the  six  areas  included  : 

in  the  set  of 

recommended 

criteria.  Particularly,  when 

those 

source  selections  that 

combined  two  criteria  into  one  area 

are  included 

in  the 

count  of  useage  of  the  recommended  : 

set  criteria 

then  their 

position  is  well  consolidated. 

The 

two  leading 

criteria  in 
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the  recommended  set.  Technical  and  Management,  were  then 
used  in  89%  and  82%  of  the  source  selections  respectively, 
and  they  lead  the  other  criteria  by  a  substantial  margin. 
Indeed,  considering  the  types  of  procurements  for  which  ASD 
would  normally  expect  to  contract  out,  one  would  suggest 
that  Technical  and  Management  are  intuitively  very  strong 
candidates  for  use  in  virtually  every  source  selection  for 
that  Division.  The  remaining  criteria  in  the  recommended 
set  are  obviously  still  common  enough  to  be  important  but 
there  are  few  criteria  outside  this  set  that  appear 
sufficiently  unique  as  to  not  be  adequately  covered  by  the 
general  criteria  within  the  recommended  and  mandatory  sets. 
For  example.  Contractor  Capability  and  Contractor  Past 
Performance,  while  being  both  extremely  important  elements 
in  an  evaluation  are  probably  more  useful  as  measures  of 
items  in  each  area  than  criteria  areas  themselves. 

The  criteria  areas  thus  reduce  to  a  very  manageable 
number  which  could  be  used  both  as  a  ’’standard"  set  of 
criteria  for  a  program  manager  to  select  from,  and  also  for 
inclusion  in  the  heuristic  model  for  the  decision  support 
system  described  in  Section  III. 

The  results,  in  summary,  are: 

Mandatory  Criteria  Set 
Cost/Price 


Recommended  Criteria  Set  (in  descending  order  of 
useage  frequency): 

Technical 

Management 

Logistics 

Operations 

Production 

Schedule 

Coproduction/Offset. 

An  examination  of  the  items  under  each  area  revealed 
a  much  greater  range,  as  would  be  expected  with  the  more 
explicit  definition  an  item  entails.  Although  many  items 
appeared,  as  was  the  case  with  the  areas,  to  have  several 
different  terms  referring  to  substantially  the  same  thing, 
any  attempt  at  substitution  and  rationalization,  as  was  done 
for  the  areas,  would  have  been  difficult  to  validate.  A 
list  of  items  under  the  heading  of  each  area  and  the  number 
of  source  selections  in  which  each  item  was  used  is  given  in 
Appendix  C,  This  appendix  should  be  read  in  conjunction 
with  Appendix  B  to  see  the  relative  rate  of  usage  of  items 
between  areas  and  within  the  dataset  as  a  whole. 

As  an  example  of  the  use  of  the  appendices  consider  the 
areas  Contractor  Capability  and  Cost/Price.  Contractor  Past 
Performance,  as  an  item  under  the  area  of  Contractor 
Capability,  was  used  three  times  in  the  six  occurrences  of 
Contractor  Capability,  or  50%  of  the  times  that  Contractor 
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Capability  was  used  as  an  area.  Completeness,  on  the  other 
hand,  was  used  43  times  as  an  item  of  Cost/Price,  but  that 
equates  to  only  43/75  or  a  57%  usage  rate.  While  some  items 
were  used  more  frequently  than  others  there  was  no  clear  set 
of  significantly  more  popular  items  as  there  was  with  the 
areas. 

Further,  with  the  very  large  range  of  items  and  the 
high  proportion  of  single  use  items,  one  can  conclude  from 
simple  visual  inspection  that  there  will  be  no  statistically 
significant  or  meaningfull  associations  between  system 
attributes  and  items.  There  is  the  possibility  that,  with 
in-depth  knowledge  of  each  system,  the  item  names  could  be 
rationalized  or  standardized  in  nomenclature  sufficiently  to 
observe  some  degree  of  association  between  items  and  their 
system's  attributes.  But  such  a  study  would  be  extremely 
involved  and  most  likely  impractical.  If  it  could  be 
achieved  then  a  third  dimension  (items  for  each  area)  could 
be  added  to  each  of  the  tables  of  the  heuristic  model 
previously  discussed. 

One  final  observation  made  from  the  examination  of 
items  was  the  use  as  specific  criteria  of  factors  referred 
to  by  AFR  70-15  as  assessment  criteria.  Again  recall  that 
the  entire  dataset  was  pre-issue  of  the  current  AFR  and  the 
more  explicit  explanations  contained  there-in  may  produce 
more  consistent  area  and  item  selection  in  the  f aluie. 

The  range  of  items  in  each  area  was  very  large  and  no 
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common  patterns  were  evident.  There  was  an  overwhelming 
majority  of  single  use  items  which;  negated  any  requirement 
to  search  for  associations  between  system  attributes  and 
criteria  items.  The  incorporation  of  items  into  a  decision 
support  system  is  therefore  considered  infeasible. 
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V.  Summary,  Conclusions,  and  Recommendations 

Background 

The  acquisition  of  a  weapon  system  will  involve  the 
procurement  by  contract  of  materiel  and  services  at  numerous 
points  during  all  phases  of  the  acquisition  cycle.  When  a 
choice  between  competing  contractors  must  be  made  a  process 
called  source  selection  is  used.  Source  selection  involves 
the  evaluation  of  proposals  submitted  by  contractors  and 
selection  of  the  offer  considered  most  advantageous  to  the 
government. 

The  evaluation  of  proposals  is  achieved  by  scoring 
each  proposal  on  the  basis  of  a  set  of  predetermined 
evaluation  criteria.  The  criteria  are  defined  and  explained 
in  a  pre-solicitation  document  called  the  Source  Selection 
Plan.  Rules  and  guidance  for  the  conduct  of  source 
selections  and  the  preparation  of  source  selection  plans  are 
contained  in  several  Federal,  Defense  and  Service 
publications.  However,  the  source  selection  evaluation 
criteria  are  required  to  be  specifically  tailored  to  the 
needs  and  characteristics  of  the  system  or  subsystem  being 
procured,  and  guidance  on  the  selection  of  appropriate 
criteria  is  vague  and  fragmented. 


Research  Problem 


The  source  selection  process  is  usually  long  and 
costly  and  the  determination  of  evaluation  criteria  is  often 
one  of  the  hardest  tasks  to  be  accomplished*  Action  is 
being  taken  to  reduce  the  duration  and  cost  of  the  source 
selection  process  by  the  introduction  of  office  automation 
and  decision  support  systems  to  some  agencies.  This 
research  is  aimed  at  further  assisting  in  the  development  of 
decision  support  systems  by  examining  the  decision  processes 
involved  in  the  determination  of  source  selection  evaluation 
criteria* 

Research  Question 

The  research  question  is  whether  a  decision  support 
system  designed  to  aid  in  the  determination  of  source 
selection  evaluation  criteria  is  feasible. 

Decision  Process 

Investigation  revealed  that  guidance  in  the 
determination  of  evaluation  criteria  comes  from  several 
sources  including  regulatory  publications,  handbooks,  past 
experience,  intuition,  and,  in  particular,  the  files  of  past 
source  selections.  In  many  cases  the  decision  as  to  which 
criteria  to  use  in  a  given  source  selection  is  a  semi- 
structured  process  which  takes  the  example  of  a  previous 
similar  source  selection  and  modifies  the  criteria  as 
necessary  and  dictated  by  the  circumstances  of  the  current 


82 


procurement.  The  potential  for  decision  support  lies  in  the 
possibility  of  aggregating  the  wealth  of  past  experience 
contained  in  the  decisions  made  to  select  criteria  for 
previous  source  selections. 

Methodology 

The  methodology  adopted  to  determine  the  feasibility 
of  a  decision  support  system  for  the  determination  of  source 
selection  evaluation  criteria  was  to  examine  a  comprehensive 
set  of  source  selection  plans  used  by  an  acquisition  agency 
on  procurements  over  about  one  decade. 

The  first  step  was  to  assess  the  extent  to  which  the 
decision  process  was  programmed  by  determining  which 
criteria,  if  any,  were  prescribed  by  regulation  and  thus 
belonged  to  a  set  of  mandatory  criteria. 

The  second  step  was  to  determine  whether  there  was 
any  relationship  between  the  character  of  the  system  being 
procured  and  the  source  selection  criteria  used  for 
contractors’  proposal  evaluation.  The  procedure  envisaged 
to  accomplish  this  was  to  describe  the  weapon  system  by 
using  a  set  of  key-word  descriptions  or  attributes  which 
collectively  define  those  characteristics  of  the  system  that 
may  influence  the  evaluation  criteria  used.  These 
attributes  of  each  system  and  the  criteria  used  by  each 
source  selection  were  then  to  be  used  to  construct  a 
contingency  table  of  cross-classified  categorical  data  which 
could  be  analyzed  with  a  commercial  computer  statistical 
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package  to  assess  the  degree  of  association  between  the 
system  attributes  and  the  criteria.  If  a  subset  could  be 
drawn  from  the  set  of  attributes  to  describe  a  new  system 
then  that  subset  could  be  used  to  indicate  the  criteria 
which  should  be  considered  for  use  in  the  source  selection 
for  the  new  system.  -  These  criteria  would  be  added  to  any 
criteria  recommended  for  consideration  by  any  other  source, 
regulations  or  handbooks,  to  form  a  set  of  recommended 
criteria. 

However,  the  author  was  unable  to  find  any  published 
methodology  for  measuring  the  degree  of  association,  if  any, 
between  the  attributes  and  the  criteria.  Conventional 
analysis  of  cross-classified  nominal  data  does  not  allow  for 
the  situation  where  a  variable  may  take  on  more  than  one 
value  for  a  given  observation  in  the  sample  population. 
Consequently,  the  author  was  driven  to  develop  a  less 
rigorous,  heuristic  methodology  for  the  basis  of  a  decision 
support  system. 

The  heuristic  is  presented  as  a  framework  upon  which 
a  decision  support  system  could  be  built.  The  method 
consists,  briefly,  of  describing  each  system  in  the 
historical  dataset  by  assigning  one  from  a  set  of  values  for 
each  of  several  variables  which  collectively  serve  to  define 
the  system  characteristics  that  affect  the  evaluation 
criteria.  For  each  of  these  variables  a  contingency  table 
is  constructed  which  can  be  analyzed  using  conventional 
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cross  tabulation  techniques  to  determine  the  degree  of 
association  between  that  system  characteristic  and  the 
evaluation  criteria.  A  new  system  may  be  described  in 
terms  of  values  of  the  attribute  variables  in  a  similar 
manner  to  that  envisaged  and  described  for  the  original 
methodology.  A  table  may  then  be  constructed  matrixing 
those  attribute  values  against  all  possible  criteria.  The 
frequency  counts  for  each  attribute  variable  value  may  be 
transferred  from  the  appropriate  contingency  table  of  the 
historical  dataset  to  the  cells  of  the  newly  constructed 
table.  The  sum  of  the  frequencies  of  each  criteria  will 
then  provide  an  indication  of  the  relative  historical  usage 
of  each  criteria  in  systems  with  those  collective  values  of 
the  attribute  variables. 

Data 

The  dataset  was  drawn  from  Source  Selection  Plans 
since  1971  for  source  selections  conducted  by  Air  Force 
Systems  Command  Aeronautical  Systems  Division  and  consisted 
of  the  evaluation  criteria  in  terms  of  area  and  item  names 
used  for  each  of  81  source  selections. 

The  author  attempted  to  define  system  attribute 
variables  and  assign  values  to  those  variables  for  each 
system  in  the  dataset,  but  found  that  the  information 
contained  within  the  plane  was  insufficient  to  complete  that 
task  to  any  reasonable  level  of  satisfaction.  Further,  due 
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to  the  circumstances  of  the  author,  the  sensitive  nature  of 
source  selection  material,  and  the  security  classification 
of  many  pertinent  details  of  new  weapon  systems,  access  to 
the  necessary  level  of  information  was  not  available.  Thus, 
a  test  of  the  heuristic  methodology  was  not  possible. 

The  research  was  therefore  directed  toward  the 
evaluation  criteria  to  assess  at  least  its  suitability  for 
incorporation  into  a  DSS  using  the  heuristic  method. 

Findings 

The  range  of  criteria  areas  used  in  the  sample 
source  selection  was  found  to  be  large;  far  too  large  to  be 
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effectively  used  in  a  DSS.  However,  a  relatively  few 
criteria  had  a  high  rate  of  usage  while  a  very  large  number 
had  a  very  low  rate  of  ussge.  Further  examination  shoved 
that  most  of  the  latter  class  of  criteria  were  synonomous  or 
very  nearly  so  with  one  of  the  criteria  in  the  high 
frequency  set.  When  the  dataset  was  modified  by 
substituting  the  most  co  ion  term  for  each  criteria  the 
number  of  different  criteria  reduced  to  about  10  which  is  a 
manageable  number  for  a  DSS.  While  the  modification  of  the 
dataset  was  a  somewhat  arbitrary  and  subjective  process,  and 
thus  open  to  some  criticism,  the  suggestion  that  a 
relatively  small  set  of  criteria  areas  to  choose  from  would 
cater  for  the  vast  majority  of  systems  is  intuitively 
appealing.  There  is  not  likely  to  be  significant 
practical  difference  between  the  use  of  slightly  different 
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nomenclature  at  the  area  level  of  criteria. 

The  range  of  items  in  each  area  was  also  large  and 

'  -  „  '  v 

no  common  patterns  were  evident.  In  the  case  of  items 

however,  substitution  of  more  common  names  for  similar  ones, 
as  was  done  for  the  areas,  could  not  be  justified.  Further, 
since  there  was  an  overwhelming  majority  of  single  incidence 
items,  there  was  no  point  in  trying  to  establish  the 
existence  of  any  associations  between  system  attributes  and 
criteria  items. 

Conclusions 

The  aim  of  the  research  was  to  determine  the 
feasibility  of  a  decision  support  system  to  aid  in  the 
determination  of  source  selection  evaluation  criteria.  The 
overall  conclusion  drawn  in  regard  to  that  research  question 
is  that,  as  far  as  the  research  was  able  to  proceed  given 
the  constraints  of  access  to  classified  information,  the 
development  of  a  decision  support  system  which  will  use  past 
source  selection  data  to  aid  in  the  determination  of  source 
selection  evaluation  criteria  for  a  new  system  is  feasible. 

The  conclusion  of  feasibility  however,  requires 
qualification.  Firstly,  the  more  rigorous  data  processing 
methodology  originally  envisaged  is  not  possible,  as 
developed,  using  currently  available  statistical  techniques. 
Secondly,  the  heuristic  methodology  as  described  is  feasible 
in  that  the  number  of  criteria  is  manageable  and  the 


processing  is  computer  programmable.  However,  the 
feasibility  of  describing  each  weapon  system  in  terms  of 
values  of  a  small  set  of  attribute  variables  has  not  been 
tested  and  will  only  be  testable  with  access  to  the  required 
level  of  information.  Further,  and  subsequent  to  the 
selection  of  attribute  variables,  the  association,  if  any, 
between  the  attribute  variables  and  evaluation  criteria  must 

•  ,  y 

be  established.  Thirdly,  the  criteria  items,  and  also 
therefore  factors  and  sub-factors,  exhibited  insufficient 
likelihood  of  association  with  attribute  variables  to  be 
considered  suitable  for  inclusion  in  the  DSS  model. 

Further  conclusions  which  were  drawn  from  the  analysis 
of  the  data  were: 

a.  Mandator?  Criteria.  Only  one  criterion  area, 
that  of  Cost/Price,  could  undeniably  be  shown 
to  belong  to  the  set  of  mandatory  criteria  as 
determined  by  reference  to  regulatory 
publications.  Past  performance,  while 
prescribed  as  a  factor  for  evaluation  in  source 
selections  (44,9),  is  considered  to  be  an 
assessment  criteria  and/or  a  general 
consideration,  but  not  a  specific  criteria  area 
of  the  type  to  be  included  here. 

b.  Recommended  Criteria.  The  criteria  areas 
Technical,  Management,  Logistics,  Operations, 
and  Production  are  candidates  for  inclusion  in 
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a  set  of  recommended  criteria  which  would 
provide  the  rows  for  each  contingency  table  for 
the  heuristic  methodology.. 

c»  System  Related  Criteria.  A  few  criteria  were 
obviously  not  of  the  general  nature  of  the 
recommended  criteria  and  thus  could  probably  be 
turned  on  or  off  dependent  on  the  values  of 
suitable  dichotomous  variables* 
Coproduction/Offset  was*  one  example  which  is 
clearly  only  going  to  be  applicable  in  a  select 
number  of  instances  and  thus  could  be  assigned 
to  s  set  of  system  related  criteria. 

The  observation  that  very  few  of  the  source 
selection  plans  in  the  dataset  used  more  than  five  criteria, 
areas  leads  to  the  conclusion  that  five  areas>  should  be  the 
maximum  used  for  new  systems  except  in  unusual 
circumstances.  In  the  same  vein,  those  areas  included,  in 
the  criteria  for  a  given  system  should  be  able  to  be  drawn 
from  a  relatively  small  (possibly  10  or  less)  set  of 
standard  criteria.  Since  the  areas  are  normally  subdivided 
into  items,  the  area  name  is  a  rather  broad  indicator  and 
the  author  concludes  that  the  following  set  would  probably 
satisfice  as  a  minimum  for  a  complete  set  of  recommended 
criteria: 

a.  Technical, 

b.  Management, 
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c.  Logistics, 

d.  Operations,  and 

e.  Production. 

Similarly*  a  set  of  system  related  criteria,  based 
on  the  observation  of  the  dataset  used  for  this  research 
would  be: 

a.  Coproduction/Offset, 

b.  Test, 

c.  Schedule,  and 

d.  Contractor  Past  Performance. 

A  conclusion  was  also  drawu  that  frequently,  plans 
had  been  prepared  by  either  reference  back  to  a  previous 
plan  or  reference  M  the  same  other  source,  that  a  previous 
plan  had  been  copied  from.  In  many  cases  there  was  the 
appearance  of  a  tendency  to  change  criteria  names  in 
nomenclature,  but  not  in  meaning,  when  preparing  new  plans 
from  old,  possibly  to  present  some  semblance  of  originality. 

Recommendations 

The  results  of  the  research  lead  to  the  major 
recommendation  that  the  research  should  be  continued  in  the 
pursuit  of  a  decision  support  system  for  the  determination 
of  source  selection  evaluation  criteria.  In  conjunction 
with  that  recommendation,  the  following  recommendations  are 
also  made: 

a.  Since  access  to  classified  data  will  be 
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involved  in  future  research,  the  impetus  and 
authority  to  conduct  the  effort  should 
originate  at  such  a  level  as  to  ensure  adequate 
access  to  the  necessary  information. 

b.  The  determination,  test,  and  validation  of 
suitable  system  attribute  variables  will  be  a 
major  undertaking  and  for  that  reason  'should  be 

■  V 

commenced  with  the  declared  aim  of  implements- 
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tion  of  a  specific  decision  support  facility  oh 
successful  completion  of  the  research. 

c.  Attention  should  be  paid  to  the  nature  of  the 

t 

final  DSS  and  its  interface  with  existing  or 

i ' 

planned  office  automation  systems  and  other 
decision  support  aids  with  the  aim  of 
constructing  the  database  for  the  research  to 
be  compatible  with  future  requirements.  The 
capabilities  of  currently  available  micro¬ 
computer  database  management  systems  will 
probably  not  meet  that  requirement  or  permit 
sufficient  growth  potential. 

d.  The  quality  of  the  historical  evaluation 
criteria  used  for  the  database  should  be 
assessed  by  measurement  of  the  success  of  the 
source  selection  process  in  each  case  to  ensure 
that  future  decisions  are  not  influenced 
adversely  by  bad  past  decisions. 
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Appendix  A 


Area  Frequencies  and  Substitutions 


Area 

Usage 

% 

Modified  Area 

ABILITY  TO  MEET  SCHEDULE 

1 

1 

SCHEDULE 

ADEQUACY  OF  DESIGN 

1 

1 

TECHNICAL 

ADEQUACY  OF  PROGRAM 

2 

2 

MANAGEMENT 

COMPLIANCE  WITH  SPECIFICATION 

I 

1 

OPERATIONS 

CONTRACTOR  CAPABILITY 

5 

6 

CONTRACTOR  CAPABILITY 

CONTRACTOR  PAST  PERFORMANCE 

6 

7 

CONTRACTOR  PAST  PERFORMANCE 

COPRODUCTION  REQUIREMENTS 

1 

1 

COPRODUCTION/ OFFSET 

COST/PRICE 

60 

74 

COST/PRICE 

DESIGN  APPROACH 

4 

4 

TECHNICAL 

DESIGN  CONCEPT 

1 

1 

TECHNICAL 

ENGINEERING 

1 

1 

TECHNICAL 

FLIGHT  TEST  EVALUATION 

I 

1 

OPERATIONS 

INTEGRATED  LOGISTICS  SUPPORT 

4 

4 

LOGISTICS 

LIFE  CYCLE  COST 

15 

18 

COST/PRICE 

LOGISTICS 

20 

24 

LOGISTICS 

LOGISTICS  CONCEPTS 

1 

1 

LOGISTICS 

LOGISTICS  SUPPORT 

1 

1 

LOGISTICS 

LOGISTICS  SUPPORTABILITY 

2 

2 

LOGISTICS 

LOGISTICS/ OPERATIONS 

1 

1 

LOGISTICS/ OPERATIONS 

LOGISTICS/PROGRAM  ADEQUACY 

1 

1 

LOGISTICS/MANAGEMENT 

LOGISTICS/ SUPPORTABILITY 

1 

1 

LOGISTICS 

LOGISTICS/ SUPPORTABILITY  PLANS 

1 

1 

LOGISTICS 

MANAGEMENT 

41 

50 

MANAGEMENT 

MANAGEMENT  &  MANUFACTURING 

3 

3 

MANAGEMENT/PRODUCTION 

MANAGEMENT  APPROACH 

3 

3 

MANAGEMENT 

MANAGEMENT  EXCELLENCE 

1 

1 

MANAGEMENT 

MANAGEMENT/KANUFACTURING 

1 

1 

MANAGEMENT/PRODUCTION 

MANAGEMENT/PRODUCTION 

1 

1 

MANAGEMENT/PRODUCTION 

MANAGEMENT/PRODUCTION  CAPABILI 

2 

2 

MANAGEMENT/PRODUCTION 

MANUFACTURING 

4 

4 

PRODUCTION 

MANUFACTURING/QUALITY  ASSURANC 

2 

2 

PRODUCTION 

OFFSET 

1 

1 

COPRODUCTION/  OFFSET 

OPERATIONAL 

8 

9 

OPERATIONS 

OPERATIONAL  CAPABILITY 

1 

1 

OPERATIONS 

OPERATIONAL  UTILITY 

6 

7 

OPERATIONS 

OPERATIONAL  UTILITY/TECHNICAL 

2 

2 

TECHNICAL/OPERATIONS 

OPERATIONAL/TECHNICAL 

1 

1 

TECHNICAL/ OPERATIONS 

OPERATIONS 

1 

1 

OPERATIONS 

OVERALL  CAPABILITY 

i 

i 

•to 

CONTRACTOR  CAPABILITY 

PERFORMANCE 

l 

1 

OPERATIONS 

PROCUREMENT 

i 

1 

MANAGEMENT 

PRODUCTION 

2 

2 

PRODUCTION 

PRODUCTION  CAPABILITY 

1 

1 

PRODUCTION 

PRODUCTION  CAPACITY 

1 

1 

PRODUCTION 

PRODUCTION/MANUFACTURING  CAPAB 

1 

1 

PRODUCTION 
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Area 

PROGRAM  ADEQUACY 

PROGRAM  ADEQUACY  &  COMPETITION 

PROGRAM  ADEQUACY/SUPPORT 

PROGRAM  IMPLEMENTATION 

PROGRAM  MANAGEMENT 

PROGRAM  MANAGEMENT/PRODUCTION 

PROJECT  IMPLEMENTATION 

READINESS  &  SUPPORT 

RISK  ASSESSMENT 

SCHEDULE 

SOFTWARE  DEVELOPMENT  CAPABILIT 

SUPPORT 

SUPPORTABILITY 

TECHNICAL 

TECHNICAL  ADEQUACY 

TECHNICAL  APPROACH 

TECHNICAL  APPROACH/OPERATIONS 

TECHNICAL  COMPETENCY 

TECHNICAL  EXCELLENCE 

TECHNICAL/MANUFACTURING 

TECHNICAL/OPERATIONAL 

TECHNICAL/OPERATIONAL  UTILITY 

TEST 

TEST  &  EVALUATION  DEMONSTRATE 
TEST-IN-CONTAINER  CAPABILITY 
USEABILITY/SUPPORTABILITY 
WEIGHT 


Usage  %  Modified  Area 

2  2  MANAGEMENT 

1  1  MANAGEMENT 

1  1  MANAGEMENT 

1  I  MANAGEMENT 

3  3  MANAGEMENT 

1  1  MANAGEMENT/PRODUCTION 

1  1  MANAGEMENT 

1  1  LOGISTICS 

I  1  OPERATIONS 

7  8  SCHEDULE 

1  1  PRODUCTION 

1  1  LOGISTICS 

7  8  LOGISTICS 

53  65  TECHNICAL 

I  I  TECHNICAL 

3  3  TECHNICAL 

1  1  TECHNICAL/OPERATIONS 

1  1  TECHNICAL 

I  1  TECHNICAL 

1  1  TECHNICAL/PRODUCTION 

1  l  TECHNICAL/OPERATIONS 

1  1  TECHNICAL/OPERATIONS 

1  I  OPERATIONS 

l  1  OPERATIONS 

1  1  OPERATIONS 

1  1  LOGISTICS 

1  1  OPERATIONS 
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Appendix  B:  Area  Usage  for  Modified  Dataset 


Area 


Usage  X 


CONTRACTOR  CAPABILITY 

6 

7 

CONTRACTOR  PAST  PERFORMANCE 

6 

•f 

I 

COPRODUCTION/OFFSET 

2 

2 

COST/PXICE 

75 

92 

LOGISTICS 

40 

49 

LOGISTICS/MANAGEMENT 

1 

1 

LOGISTICS/OPERATIONS 

1 

1 

MANAGEMENT 

57 

70 

MANAGEMENT/PRODUCTION 

8 

9 

OPERATIONS 

24 

29 

PRODUCTION 

12 

14 

SCHEDULE 

8 

9 

TECHNICAL 

66 

81 

TECHNICAL/OPERATIONS 

6 

7 

TECHNICAL/PRODUCTION 

1 

1 

Appendix  C:  Item  Usage  for  Modified  Dataset 

Area:  CONTRACTOR  CAPABILITY 


Item  Usage 

ABILITY  1 
COMMITMENT  1 
CONTRACTOR  PAST  PERFORMANCE  3 
EXPERIENCE  3 
FACILITIES  1 
FINANCIAL,  PLANT  &  EQUIPMENT  2 
FUNCTIONAL  OPERABILITY  1 
MANAGEMENT  CONTROL  SYSTEMS  3 
PROJECT  ORGANIZATION  1 
PERFORMANCE  1 
PERSONNEL  3 
PROJECT  MANAGEMENT  1 
PROPULSION  INTEGRATION  I 
STRUCTURAL  INTEGRITY  1 


Area:  CONTRACTOR  PAST  PERFORMANCE 
No  items  used  for  this  area. 
Area:  COPRODUCTION/OFFSET 


Item  Usage 

CO-PRODUCTION  PLAN  1 

CO-PRODUCTION  STRUCTURE  1 

OFFSET  REQUIREMENTS  1 

Area:  COST/PRICE 

Item  Usage 

ACQUISITION  COST  8 

ACQUISITION  COST/PRICE  2 

APPROACH  TO  COSTING  NEXT  PHASE  1 

COMPARATIVE  ANALYSIS  6 

COMPLETENESS  43 

CONTRACTOR  PAST  PERFORMANCE  1 

COST  ANALYSIS  4 

COST  TRADE-OFF  ANALYSES  1 

FINANCIAL  RESPONSIBILITY  3 

FULL  SCALE  ENGINEERING  DEVELOP  2 

LEVEL  OF  RISK  1 
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Item  Usage 

LIFE  CYCLE  COST  12 
LOGISTICS  SUPPORT  COST  1 
LOGISTICS  SUPPORT  COST  MODEL  1 
MOST  PROBABLE  COST  17 
OPERATING  &  SUPPORT  COST  1 
OPERATIONS  &  SUPPORT  COST  2 
OPTION  COSTS  1 
PAST  PERFORMANCE  2 
PAST  PERFORMANCES  1 
PRICE  ANALYSIS  3 
PRODUCTION  COSTS  2 
PROGRAM  COST  1 
RDT&E  COSTS  2 
REALISM  1 
REASONABLENESS  1 
REASONABLENESS  &  REALISM  51 
RESPONSIVENESS  TO  THE  RFP  2 
RISK  4 
SUPPORT  COST  1 
SUPPORT  EQUIPMENT  COSTS  1 
TRACEABILITY  3 
TRACKABILITY  1 
WEAPON  SYSTEM  COST  1 


Area:  LOGISTICS 

Itea  Usage 

ADEQUACY  OF  SUPPORT  3 

AVAILABILITY  1 

CAPABILITY  1 

COMMONALITY  2 

COMPATIBILITY  WITH  SYSTEM  DESI  1 

COMPLIANCE  WITH  REQUIREMENT  1 

CONTRACTOR  LOGISTICS  SUPPORT  1 

CONTRACTOR  SUPPORT  1 

DESIGN  CHARACTERISTICS  1 

ENERGY  MANAGEMENT  1 

EXPERIENCE  1 

EXPERTISE/CAPABILITY  1 

FACILITIES  1 

FIELD  SUPPORT  PLANNING  6 

GROUND  SUPPORT  EQUIPMENT  1 

GROUND  SUPPORT  EQUIPMENT  PLAN  1 

IDENTIFIED  RESOURCES  1 

ILS  ORGANIZATION  1 

ILS  PLANNING  APPROACH  1 

ILS  REQUIREMENTS  1 

INTEGRATED  LOGISTIC  SUPPORT  1 
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lies  Usage 

INTEGRATED  LOGISTICS  SUPPORT  4 
INTEGRATED  SUPPORT  PLAN  4 
INTEGRATED  SUPPORT  PLANNING  3 
LIFE  CYCLE  COST  5 
LOGISTIC  SUPPORT  ANALYSIS  3 
LOGISTIC  SUPPORT  PLAN  1 
LOGISTICAL  SUPPORT  REQUIREMENT  I 
LOGISTICS  CONCEPT  1 
LOGISTICS  DESIGN  APPROACH  2 
LOGISTICS  DESIGN  CONSIDERATION  1 
LOGISTICS  DOCUMENTATION  1 
LOGISTICS  ENGINEERING/SUPPORT  1 
LOGISTICS  INTEGRATION  1 
LOGISTICS  MANAGEMENT  l 
LOGISTICS  ORGANIZATION  1 
LOGISTICS  ORGANIZATION/PERSONN  1 
LOGISTICS  PLANNING  1 
LOGISTICS  SUPPORT  ANALYSIS  5 
LOGISTICS  SUPPORT  ANALYSIS  REC  1 
LOGISTICS  SUPPORT  PLAN  1 
LOGISTICS  SUPPORT  PLANNING  1 
MAINTAINABILITY  2 
MAINTENANCE  CONCEPTS  3 
MAINTENANCE  PLAN  1 
MAINTENANCE  PLANNING  I 
MAINTENANCE  REQUIREMENTS  1 
MAINTENANCE  SUPPORT  I 
MAINTENANCE/MAINTAINABILITY  2 
MANAGEMENT  INFORMATION  SYSTEM  1 
OPERATIONAL  MAINTAINABILITY  I 
OPERATIONAL  SUPPORTABILITY  1 
PLANS  1 
PREOPERATIONAL  SUPPORT  3 
PREOPERATIONAL  SUPPORT  PLAN  1 
PROVISIONING  MANAGEMENT  I 
PROVISIONING  PLANNING  1 
QUALITY  ASSURANCE  1 
RELIAB/MAINTAINABILITY  FORECAS  2 
RELIABILITY  1 
RELIABILITY  &  LONGEVITY  1 
RELIABILITY  &  MAINTAINABILITY  5 
RELIABILITY  FORECASTS  1 
SE  DEVELOPMENT  PLANNING  1 
SPARE/REPAIR  PARTS  PLAN  1 
SUPPLY  SUPPORT  2 
SUPPLY  SUPPORTFIELD  SUPPORT  PL  1 
SUPPORT  EQUIPMENT  6 
SUPPORT  EQUIPMENT  ANALYSIS  1 
SUPPORT  EQUIPMENT  DEVELOPMENT  X 
SUPPORT  EQUIPMENT  PLANNING  1 
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Item  Usage 

SUPPORT  EQUIPMENT  REQUIREMENTS  1 

SUPPORT  REQUIREMENTS  2 

SUPPORT ABILITY  3 

SYSTEM  AVAILABILITY  1 

SYSTEM  FLEXIBILITY  1 

SYSTEM  PERFORMANCE  1 

TECHNICAL  APPROACH  1 

TECHNICAL  DATA  4 

TECHNICAL  MANUAL  REQUIREMENTS  1 

TECHNICAL  MANUALS  1 

TRAINING  5 

TRANSITION  TO  ORGANIC  SUPPORT  1 

Area;  LOGISTICS/MANAGEMENT 

Item  Usage 

PATA  &  CONFIGURATION  MANAGEMEN  1 

INTEGRATED  LOGISTICS  SUPPORT  1 

MANUFACTURING/QUALITY  ASSURANC  1 

RELIABILITY  &  MAINTAINABILITY  1 


Area:  LOGISTICS/OPERATIONS 

No  item  used  for  this  area. 
Area:  MANAGEMENT 


Item  Usage 

ABILITY  TO  MEET  SCHEDULES  1 
ADEQUACY  OF  PLANNING  1 
ADMINISTRATION  I 
APPROACH  &  ORGANIZATION  8 
COMPLETENESS  OF  PLANNING  1 
CONFIGURATION  MANAGEMENT  11 
CONFIGURATION  MANAGEMENT  PLAN  3 
CONFIGURATION/DATA  MANAGEMENT  1 
CONTRACTOR  CAPABILITY  1 
CONTRACTOR  PAST  PERFORMANCE  3 
CONTROL  OF  COSTS  5 
CONTROL  PROCEDURES  1 
COST  MANAGEMENT  SYSTEM  1 
COST  RISK  ASSESSMENT  1 
CRITICAL  PATH  SCHEDULE  1 
DATA  &  CONFIGURATION  MANAGEMEN  3 
DATA  MANAGEMENT  9 
DEMONSTRATED  ABILITY  1 
DESIGN  TO  COST  PLAN  1 
DUAL  SOURCING/BREAKOUT  1 
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Item 


Usage 


ENGINEERING  MANAGEMENT  1 
ENGINEERING  PLANNING  1 
INFORMATION  MANAGEMENT  4 
INTEGRATED  LOGISTICS  SUPPORT  1 
INTEGRATED  PROGRAM  SCHEDULES  1 
LABOUR  RELATIONS  1 
LOGISTICS  SUPPORT  REQUIREMENTS  1 
MANAGEMENT  4 
MANAGEMENT  &  ORGANIZATION  2 
MANAGEMENT  APPROACH  1 
MANAGEMENT  CAPABILITY  1 
MANAGEMENT  INFORMATION  SYSTEM  1 
MANAGEMENT  INTERFACE  1 
MANAGEMENT  METHODS  1 
MANAGEMENT  OF  SUPPORT  FUNCTION  1 
MANAGEMENT  PLANNING  &  ORGANIZA  1 
MANAGEMENT  PROCEDURES  1 
MANAGEMENT  STRUCTURE  4 
MANAGEMENT  VISIBILITY  1 
MANNING  1 
MANUFACTURING  1 
MANUFACTURING  CAPABILITY  1 
MANUFACTURING/OPERATIONS  1 
MANUFACTURING/PRODUCTION  PLANN  1 
MANUFACTURING/QUALITY  ASSURANC  6 
O&S  COST  ASSESSMENT  1 
ORGANIZATION  4 
ORGANIZATION  &  PLAN  1 
ORGANIZATION  STRUCTURE  3 
ORGANIZATION/PERSONNEL  1 
ORGANIZATIONAL  STRUCTURE  1 
OTHER  MANAGEMENT  FACTORS  1 
OVER  AND  ABOVE  TASK  ORDERS  1 
OVERALL  PROJECT  ORGANIZATION  1 
PARTS  CONTROL  PROGRAM  1 
PAST  PERFORMANCE  1 
PERSONNEL  8 
PROCEDURES  1 
PRODUCIBILITY  1 
PRODUCTION  CAPABILITY  4 
PRODUCTION  MANAGEMENT  1 
PRODUCTION  PLANNING  5 
PROGRAM  CONTROL  2 
PROGRAM  MANAGEMENT  5 
PROGRAM  MANAGEMENT  ORGANIZATIO  I 
PROGRAM  MANAGEMENT  PLAN  2 
PROGRAM  MANAGEMENT  PLANNING  3 
PROGRAM  MANAGEMENT  SYTSTEM  1 
PROGRAM  MANAGER  1 
PROGRAM  PLAN  1 
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Item 


Usage 


PROGRAM  PLANS/SCHEDULES  1 
PROGRAM  PLANS/SCHEDULES/CAPABI  1 
PROGRAM  REVIEWS  1 
PROGRAM  SCHEDULES  2 
PROJECT  MANAGEMENT  3 
PROJECT  MANAGEMENT  EXPERIENCE  3 
PROJECT  OFFICE  ESTABLISHMENT  2 
PROJECT  ORGANIZATION  1 
PROPOSAL  COST  EVALUATION  1 
PROPOSED  WORK  BREAKDOWN  STRUCT  1 
PROVIDE  &  MANAGE  ALL  RESOURCES  1 
QUALITY  ASSURANCE  7 
QUALITY  ASSURANCE  SYSTEM  1 
QUALITY  ASSURANCE/R&M  1 
RELIABILITY  &  MAINTAINABILITY  1 
RESOURCE  PLANNING  1 
RESOURCES  1 
RESOURCES  &  MANUFACTURING  CAPA  1 
RESOURCES/MANUFACTURING  CAPABI  -  1 
SCHEDULE  2 
SCHEDULES  8 
SCHEDULES  &  COSTS  1 
SPECIFIC  EXPERIENCE  1 
SUBCONTRACT  MANAGEMENT  6 
SUBCONTRACTOR  1 
SUBCONTRACTOR  MANAGEMENT  1 
SUPPORT  PLANS  1 
SYSTEM  SAFETY  2 
SYSTEM  TEST  PLAN  2 
SYSTEM  TEST  PLANNING  2 
SYSTEMS  ENGINEERING  MANAGEMENT  1 
TECHNICAL  1 
TEST  &  EVALUATION  I 
VISIBILITY  &  CONTROL  SYSTEMS  1 

Area:  MANAGEMENT/PRODUCTION 

Item  Usage 

CO-PRODUCTION  1 
CONFIGURATION  MANAGEMENT  3 
DATA  MANAGEMENT  2 
ENGINEERING  MANAGEMENT  1 
EXPERIENCE  1 
MANAGEMENT  APPROACH  2 
MANAGEMENT  CONCEPT  1 
MANAGEMENT  METHODS  1 
MANAGEMENT  SUPPORT  PLANS  1 
MANUFACTURING  2 
PRODUCTION  CAPABILITY  1 
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Item  Usage 

PRODUCTION  CAPABILITY/REQUIREM  2 

PRODUCTION  MANAGEMENT  I 

PROGRAM  MANAGEMENT  2 

PROGRAM  MANAGEMENT  ORGANIZATIO  1 

PROGRAM  MANAGEMENT  PLAN  1 

PROGRAM/PRODUCTION  MANAGEMENT  1 

QUALITY  ASSURANCE  1 

SYSTEM  CONFIGURATION  CHANGES  2 

Area:  OPERATIONS 

Item  Usage 

ABILITY  TO  MEET  REQUIREMENTS  -  1 

AVAILABILITY  2 

AVIONICS  GROUND  OPERATIONS  1 

COMPLIANCE  WITH  RFP  1 

CONCEPTUAL  UNDERSTANDING  1 

CONTRACTOR  TEST  RESULTS  1 

DESIGN  1 

DESIGN  APPROACH  1 

ENROUTE  PERFORMANCE  2 

FLIGHT  TEST  PROGRAM  1 

GOVERNMENT  LABORATORY  TESTING  1 

GROUND  OPERATIONS  1 

GROUND  TEST  PROGRAM  1 

INFLIGHT  PERFORMANCE  1 

MISSION  PERFORMANCE  2 

MISSION  PRODUCTIVITY  1 

MISSION  SCENARIOS  &  THREAT  1 

MISSION  SUITABILITY  1 

OPERATIONAL  CONCEPTS  1 

OPERATIONAL  CONSTRAINTS  1 

OPERATIONAL  PERFORMANCE  1 

OPERATIONAL  SAFETY  2 

OVERALL  EFFECTIVENESS  1 

PERFORMANCE  3 

PRODUCTIVITY  I 

QUALITY  ASSURANCE  1 

RELIABILITY  &  MAINTAINABILITY  6 

SCENARIO  SOLUTIONS  1 

SEARCH  &  RESCUE  PHASE  1 

SOFTWARE  VERIFICATION  1 

SOUNDNESS  OF  APPROACH  1 

SUBSYSTEM/SYSTEM  INTEGRATION  1 

SUPPORTABILITY  3 

SURVIVABILITY  &  VULNERABILITY  1 

SYSTEM  CAPABILITY  1 

SYSTEM  OPERATIONAL  CONCEPT  1 

TECHNICAL  PROPOSALS  1 
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Item 

TECHNICAL  PUBLICATIONS' 
TEST  PLANNING 


Usage 


•  -U:  .  *  1 
'  *■  '  1 


Area:  PRODUCTION 


Item 


ABILITY  TO  MEET  PRODUCTION  REQ 
ABILITY  TO  MEET  SCHEDULES 
BUSINESS  PLAN 

CRITICAL/LONG  LEAD  TIME  ITEMS 

ENERGY  EFFECTIVENESS 

FACILITIES 

MANAGEMENT  SYSTEM 

■MANAGEMENT  SYSTEMS  &  CONTROL 

MANUFACTURING  CAPABILITY 

MANUFACTURING  CAPACITY/CAPABIL 

MANUFACTURING  PLANNING 

MANUFACTURING  TECHNOLOGY 

ORGANIZATION 

ORGANIZATION  STRUCTURE 

PERSONNEL 

PRODUCIBILITY 

PRODUCTION  CAPABILITY 

PRODUCTION  CAPABILITY  REVIEW 

PRODUCTION  MANAGEMENT  SYSTEM 

PRODUCTION  PLAN 

PRODUCTION  PLANNING 

PROGRAM  PLANNING 

QUALITY  ASSURANCE 

REALISM/RISKS 

RESOURCES 

SCHEDULING  &  CONTROL  PROCEDURE 
SOFTWARE  DEVELOPMENT  TOOLS 
SOFTWARE  MANAGEMENT  APPROACH 
SOFTWARE  MANAGEMENT  TOOLS 
SOFTWARE  PERSONNEL  RESOURCES 
SUBCONTRACT  ARRANGEMENTS 
SUBCONTRACTING 

SUBCONTRACTOR/MATERIAL  MANAGEM 
TOTAL  MANAGEMENT  APPROACH 
UNDERSTAND  POTENTIAL  PROBLEMS 


Usage 


Area:  SCHEDULE 


Item 


ABILITY  TO  MEET  SCHEDULE 
COMPLETENESS  &  REALISM 
CONFIGURATION  &  DATA  MANAGEMEN 


Usage 
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AftMfctfcS, 


sgaHafflssaasHsssal 


Item  Usage 

INTEGRATION  WITH  LATER  PHASES  ?,•  •v-l 

ORGANIZATION  STRUCTURE  v- •  v*  2 

PERSONNEL  4  ,  ,  r  ,l 

PLANNING  .-•■•!  -.jvi-’!  2 

PRODUCTION  PLANNING  >  ■■:'  -;4‘. 

Area:  TECHNICAL  ;•  i 

..  ;  .  •"  •*"  x.  '.a  ' 

Item  v  .  ijsage 


AEROMECHANICS  •■•■....  •>  *4 
AIR  CONDITIONER  CART  ,/  ■  ;.v>?  <  -  <.  *  1 
AIR  FORCE  REVIEW  ,  1 
AIR  VEHICLE  DEVELOPMENT  t  .  1 
AIRCRAFT  INSTALLATION  1 
AIRCRAFT  INSTALLATION  CONSIDER  1 
AIRCRAFT  INTEGRATION/INTERFACE  ‘0.  2 
AIRCRAFT  INTERFACES  •-  2 
AIRCRAFT  STRUCTURAL  INTEGRITY  1 
AIRFRAME  ,  2 
AVIONICS  INTEGRATION  1 
BIRD  STRIKE  1 
CAPABILITY  OF  SYSTEM  \  ;1 
CART  :  .  v  1 
COMMONALITY  •  -  <1. 
COMPLIANCE  WITH  RFP  '  14 
COMPUTATION  SYSTEM  1 
COMPUTER  PROGRAMS  ..  1 
COMPUTER  RESOURCES  1 
CONCEPTUAL  UNDERSTANDING  .9 
CONFIGURATION  DESIGN  1 
CONFIGURATION  DESIGN  FEATURES  1 
CONSOLES  &  DISPLAYS  1 
CONTRACTOR  PAST  PERFORMANCE  2 
COST  CONSCIOUS  APPROACH  •  1 
CREW  SYSTEMS  1 
D/V  TRANSITION  TO  FULL  SCALE  D  ,1 
DATA  BASE  MANAGEMENT  SYSTEM  1 
DATA  PROCESSING  HARDWARE  1 
DEGREE  OF  RISK  1 
DESIGN  12 
DESIGN  APPROACH  2 
DESIGN  CONCEPT  2 
DESIGN  REQUIREMENTS  1 
DOCUMENTATION  1 
DURABILITY  1 
EASE  OF  MODIFICATION  1 
EMULATION/SIMULATION  SUBSYSTEM  1 
ENGINE  1 
ENGINEERING  CAPABILITY  1 
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Item 


Usage 


EXPERIENCE  "  ' 

EXPERIENCE  &  ORGANIZATION 
EXPERIENCE  &  PERSONNEL 
FLIGHT  CONTROLS 
FLIGHT  EQUIPMENT 
FLIGHT  STABILITY  &  CONTROL 
FUNCTIONAL  PROGRAMS 
GENERAL  KNOWLEDGE 
GENERATOR  CART 
HARDWARE 

-HARDWARE  DESIGN  &  PERFORMANCE 
IN-DEPTH  KNOWLEDGE  - 

INGENUITY  &  IMAGINATION 
INTEGRATION 
INTEGRATION/INTERFACE 
LANDING  GEAR  DESIGN 
MAINTAINABILITY 
MANUFACTURING  CAPABILITY 
MANUFACTURING  DESIGN 
MANUFACTURING  MANAGEMENT 
MATERIALS/TECHNIQUES/EQUIPMENT 
MISSILE  SYSTEM 
MOTION  SYSTEM 
OPERABILITY 

OPERATIONAL  TESTING  &  COMPATIB 
OPERATIONS  METHODOLOGY 
OPERATIONS/MAINTENAN  3E  REQTS 
OPTICS 

ORGANIZATIONAL  STRUCTURES 
OVERALL  DESIGN  APPROACH 
PAST  EXPERIENCE 
PAST  PERFORMANCE 
PERFORMANCE 

PERFORMANCE  ASSESSMENT 
PERFORMANCE  VERIFICATION 
PERSONNEL 

PHASE  I  TEST  RESULTS 
PHASE  II  APPROACH 
PRODUCTION  DESIGN 
PROPULSION  &  POWER 
PROPULSION  &  VEHICLE  POWER 
PROPULSION  INTEGRATION 
QUALITY  ASSURANCE 
RELIABILITY 

RELIABILITY  &  MAINTAINABILITY 
RESOURCES 

RESULTS  OF  FSED  CONTRACT 

RISK 

SAFETY 

SHELTERIZATION 


-  3 

i 

."i 

1 

i 

i 

3 

i 

1 

r 

:  i 

;;  4 
s  ?i 
■v  M 

!  1 
1 
1 

-  1 
-I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 
6 
1 
2 

4 
1 
1 
1 
1 
1 
1 
1 
1 
2 
1 
1 
4 
1 
1 
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Item 


Usage 


SIMPLICITY 
SIMULATION  FIDELITY 

SOFTWARE  '  .  O 

SOFTWARE  DESIGN  &  DEVELOPMENT  -*•-  *t-  ■  i 

SOUNDNESS  OF  APPROACH 
SOUNDNESS  OF  DESIGN 
SPECIAL  CONSIDERATIONS  >. .. 

SPECIAL  TECHNICAL  FACTORS 
SPECIFICATION  COMPLIANCE  : 

SPECIFICATIONS  - 

STRUCTURAL  DESIGN 

STRUCTURAL  INTEGRITY  '  ' 

STRUCTURE  &  LANDING  GEAR  DESIG 
SUBSYSTEM/SYSTEM  INTEGRATION 
SUPPORT  EQUIPMENT 

SUPPORT ABILITY  - 

SYSTEM  CONCEPT 

SYSTEM  DESIGN 

SYSTEM  DESIGN/PERFORMANCE 

SYSTEM  ENGINEERING  APPROACH 

SYSTEM  HARDWARE 

SYSTEM  INTEGRATION 

SYSTEM  PERFORMANCE 

SYSTEM  REQUIREMENTS 

SYSTEM  RISK  REDUCTION 

SYSTEM  SOFTWARE 

SYSTEM  TECHNICAL  PERFORMANCE 

SYSTEM  TEST  &  EVALUATION 

SYSTEMS  ENGINEERING  APPROACH 

SYSTEMS  ENGINEERING  MANAGEMENT 

TECHNICAL  APPROACH 

TECHNICAL  DEVELOPMENT 

TECHNICAL  RISK 

TECHNIQUES  &  METHODS 

TECHNOLOGY  BASE 

TEST 

TEST  &  EVALUATION 
TESTING 

TRADE-OFF  EVALUATION 
TRAINING 

UNDERSTANDING  OF  THE  PROBLEM 

USABILITY 

VISUAL  SYSTEM 

WEAPON  INTERFACES 

WEARLIFE 

WORK  CONTROLS 


1; 


1 

7 

1 

1 

1 

2 

3 

1 

1 
1 
1 

2 
1 
1 
1 
2 
1 
5 

7 
1 
1 
1 
1 
1 
5 
1 
1 
1 

8 
1 
1 
1 
1 
1 
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Area;  TECHNICAL/OPERATIONS 


i  Item  —  '  Usage 

AIR  VEHICLE  ‘  :  •  '•  2 

aircrew  Accommodations  •  j  i 

avionics  *  :  •  3 

CABIN  CONFIGURATION  : ■  l 

CARGO  COMPARTMENT  ’  1 

COCKPIT  CONFIGURATION  • ^  ;  1 

COMMONALITY  ‘  '  Y  v'  ‘  *  i 

COMPUTATIONAL  SYSTEM  -  1 

DESIGN  QUALITY  1 

ENGINEERING  SPECIALITIES  1 

FLIGHT  STATION  "  "'■  r  ‘  'M 

INFLIGHT  PERFORMANCE  '  '  1 

OPERATIONAL  CAPABILITY  1 

QUALIFICATION  TESTING  '  1 

RELIABILITY  &  MAINTAINABILITY  1 

SPECIAL  MISSION  EQUIPMENT  1 

STUDENT  STATION/INSTRUCTIONAL  1 

TEST  &  EVALUATION  1 

UNDERSTANDING  OF  THE  PROBLEM  1 

USABILITY  1 

VISUAL  SYSTEMROACH  1 

Area:  TECHNICAL/PRODUCTION 

-  >*  "i  >’  * 

Item  Usage 

FACILITIES  1 

PERSONNEL  1 

PRODUCTION  PLANNING  1 

QUALITY  ASSURANCf  1 
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